Nov 02 17:22:19.717053 ip-10-0-128-71 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:22:20.058282 ip-10-0-128-71 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:22:20.058282 ip-10-0-128-71 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:22:20.058282 ip-10-0-128-71 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:22:20.058282 ip-10-0-128-71 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:22:20.062870 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062758 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.062870 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062870 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062876 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062880 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062883 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062889 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062891 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062894 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062897 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062899 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062902 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062904 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062907 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062909 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062912 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062916 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062920 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062923 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062933 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.062941 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062936 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062938 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062941 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062944 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062947 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062949 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062952 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062955 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062958 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062961 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062963 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062966 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062968 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062971 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062973 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062975 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062978 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062981 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062984 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.063380 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062986 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062989 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062991 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062994 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062996 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.062999 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063004 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063007 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063009 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063012 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063014 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063017 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063019 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063021 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063025 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063028 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063031 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063034 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063037 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.063838 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063039 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063042 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063045 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063047 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063050 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063054 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063056 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063059 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063061 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063064 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063066 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063069 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063071 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063074 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063077 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063079 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063082 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063084 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063087 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063089 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063092 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.064313 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063094 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063097 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063100 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063102 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063105 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063110 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063113 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063115 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063118 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063120 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063123 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063126 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063128 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063131 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063134 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063136 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063139 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063141 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063144 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063146 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.064876 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063149 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063151 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063154 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063156 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063159 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063162 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063957 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063965 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063968 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063971 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063974 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063977 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063980 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063983 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063985 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063988 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063991 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063995 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.063998 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.065346 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064001 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064004 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064006 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064009 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064012 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064015 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064017 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064020 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064023 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064025 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064028 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064032 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064036 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064039 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064043 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064045 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064048 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064051 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064053 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.065815 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064056 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064059 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064061 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064064 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064066 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064068 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064071 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064074 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064076 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064079 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064081 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064086 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064088 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064091 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064094 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064096 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064099 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064101 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064104 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064107 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.066262 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064109 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064111 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064114 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064117 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064119 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064122 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064124 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064127 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064129 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064134 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064138 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064141 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064144 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064147 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064149 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064152 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064154 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064157 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064159 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064162 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.066765 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064164 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064167 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064170 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064173 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064177 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064179 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064182 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064184 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064187 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064190 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064192 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064197 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064200 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064202 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064205 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064207 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064210 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064213 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064215 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.067251 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064217 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064220 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064222 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064225 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064227 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064229 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064232 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064234 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064238 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064240 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064242 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064245 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064248 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064250 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064755 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064763 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064766 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064769 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064772 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064775 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.067728 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064778 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064781 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064783 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064786 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064788 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064791 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064794 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064797 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064799 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064802 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064804 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064807 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064809 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064815 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064817 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064820 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064822 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064825 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.068213 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064827 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064830 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064832 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064835 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064837 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064840 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064843 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064846 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064849 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064852 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064855 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064858 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064860 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064863 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064865 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064868 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064870 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064873 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064875 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.068676 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064879 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064881 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064884 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064887 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064890 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064892 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064895 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064897 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064900 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064903 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064905 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064909 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064913 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064916 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064920 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064922 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064925 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064927 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064930 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064933 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.069182 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064935 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064939 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064942 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064944 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064947 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064949 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064952 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064954 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064956 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064959 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064962 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064964 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064967 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064969 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064971 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064974 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064977 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064979 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064981 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064984 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.069702 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064986 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064989 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064991 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064993 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064996 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.064998 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065000 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065003 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065005 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065008 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065010 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065012 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065016 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065018 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065021 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065024 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065026 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065029 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065031 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.070291 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065034 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065038 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.065042 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065750 2578 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065762 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065769 2578 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065773 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065778 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065781 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065786 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065790 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065793 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065796 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065800 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065803 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065808 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065811 2578 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065815 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065818 2578 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065821 2578 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065825 2578 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065830 2578 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065833 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:22:20.070801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065836 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065840 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065843 2578 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065848 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065851 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065854 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065857 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065860 2578 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065863 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065866 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065869 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065872 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065876 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065879 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065882 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065885 2578 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065888 2578 flags.go:64] FLAG: --enable-server="true" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065891 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065895 2578 flags.go:64] FLAG: --event-burst="100" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065899 2578 flags.go:64] FLAG: --event-qps="50" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065902 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065905 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065908 2578 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065912 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:22:20.071341 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065916 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065920 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065923 2578 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065926 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065929 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065932 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065935 2578 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065938 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065941 2578 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065944 2578 flags.go:64] FLAG: --feature-gates="" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065948 2578 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065951 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065962 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065965 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065968 2578 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065971 2578 flags.go:64] FLAG: --help="false" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065974 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065978 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065981 2578 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065983 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065987 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065990 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065993 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:22:20.071948 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065996 2578 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.065998 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066001 2578 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066004 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066007 2578 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066010 2578 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066013 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066016 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066019 2578 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066024 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066027 2578 flags.go:64] FLAG: --lock-file="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066031 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066034 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066036 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066042 2578 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066045 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066048 2578 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066051 2578 flags.go:64] FLAG: --logging-format="text" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066055 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066058 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066061 2578 flags.go:64] FLAG: --manifest-url="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066064 2578 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066069 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066073 2578 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066077 2578 flags.go:64] FLAG: --max-pods="110" Nov 02 17:22:20.072550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066080 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066083 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066086 2578 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066089 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066091 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066094 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066097 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066105 2578 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066109 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066111 2578 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066114 2578 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066117 2578 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066120 2578 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066123 2578 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066126 2578 flags.go:64] FLAG: --port="10250" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066129 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066133 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e5ee1eb0fcde3293" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066137 2578 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066141 2578 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066144 2578 flags.go:64] FLAG: --register-node="true" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066147 2578 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066151 2578 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066154 2578 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066156 2578 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066159 2578 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066163 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:22:20.073246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066166 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066169 2578 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066172 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066175 2578 flags.go:64] FLAG: --runonce="false" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066180 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066183 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066187 2578 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066190 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066193 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066197 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066200 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066203 2578 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066206 2578 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066209 2578 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066212 2578 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066215 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066218 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066221 2578 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066224 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066229 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066232 2578 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066235 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066262 2578 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066266 2578 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066271 2578 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:22:20.073926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066274 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066277 2578 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066281 2578 flags.go:64] FLAG: --v="2" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066289 2578 flags.go:64] FLAG: --version="false" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066293 2578 flags.go:64] FLAG: --vmodule="" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066298 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066301 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066456 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066460 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066464 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066466 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066469 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066472 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066475 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066477 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066480 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066483 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066486 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066488 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066491 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066493 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.074535 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066496 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066498 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066501 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066504 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066506 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066509 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066511 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066513 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066517 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066519 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066524 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066526 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066529 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066531 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066534 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066536 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066539 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066541 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066545 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.075064 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066549 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066552 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066555 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066558 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066560 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066563 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066566 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066568 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066571 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066573 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066576 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066578 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066580 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066583 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066586 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066588 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066590 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066594 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066597 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.075524 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066600 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066603 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066605 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066608 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066612 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066614 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066617 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066619 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066621 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066624 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066626 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066630 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066632 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066635 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066637 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066639 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066642 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066645 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066647 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.076003 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066650 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066653 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066655 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066658 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066660 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066662 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066665 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066667 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066670 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066672 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066675 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066677 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066679 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066682 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066684 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066687 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066690 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066694 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066697 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066700 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.076453 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066702 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066705 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066707 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066710 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066712 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066714 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066717 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066719 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066722 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066725 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066727 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066743 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066746 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066749 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066752 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.066779 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:22:20.077037 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066817 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066821 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066825 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066828 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066830 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066833 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066835 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066838 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066840 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066843 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066845 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066848 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066851 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066855 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066857 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066860 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066862 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066865 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066867 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066870 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.077411 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066872 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066875 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066877 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066880 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066882 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066885 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066889 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066894 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066897 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066900 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066903 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066906 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066908 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066910 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066913 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066915 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066917 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066920 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066922 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.077911 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066925 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066927 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066930 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066932 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066935 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066940 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066945 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066948 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066950 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066953 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066955 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066958 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066960 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066962 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066965 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066967 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066970 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066972 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066975 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.078365 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066977 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066979 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066982 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066985 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066988 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066992 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066994 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066997 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.066999 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067002 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067004 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067007 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067009 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067012 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067014 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067016 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067019 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067021 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067025 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.078915 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067029 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067032 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067035 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067037 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067039 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067042 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067044 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067047 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067049 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067052 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067054 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067056 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067059 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067061 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067064 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067066 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067069 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067071 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067074 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067077 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.079385 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067079 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067082 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067084 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067086 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067089 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067092 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067095 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067097 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.067100 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.074156 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.074181 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.074197 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.074203 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074265 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074271 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074275 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074277 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074280 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074284 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.080000 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074287 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074291 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074295 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074299 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074303 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074305 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074308 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074311 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074314 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074316 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074319 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074321 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074324 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074327 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074329 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074332 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074334 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074336 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074339 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.080501 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074341 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074344 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074347 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074350 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074354 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074356 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074359 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074362 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074364 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074366 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074369 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074371 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074374 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074376 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074378 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074381 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074383 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074386 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074388 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.081030 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074390 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074393 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074395 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074398 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074400 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074402 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074405 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074408 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074411 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074413 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074417 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074420 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074423 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074426 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074429 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074431 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074434 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074437 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074440 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074442 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.081495 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074445 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074447 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074449 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074452 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074455 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074458 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074461 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074463 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074465 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074468 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074470 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074472 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074475 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074477 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074480 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074482 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074485 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074487 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074490 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.082240 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074492 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074495 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074497 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074499 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074502 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074504 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074507 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074509 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074512 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074514 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074517 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074520 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074523 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074525 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074528 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074530 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074533 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074535 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074538 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074541 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.082973 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074543 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074546 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074549 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.074583 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074752 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074758 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074761 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074764 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074766 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074769 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074771 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074775 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074777 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074780 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074783 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074785 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.083764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074788 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074791 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074793 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074796 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074798 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074801 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074803 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074806 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074809 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074812 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074815 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074817 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074820 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074823 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074825 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074828 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074830 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074833 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074835 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074837 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.084237 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074840 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074842 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074845 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074847 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074850 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074852 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074854 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074857 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074860 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074862 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074864 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074867 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074870 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074872 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074876 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074880 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074883 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074885 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074888 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.084805 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074891 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074894 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074896 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074905 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074907 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074910 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074913 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074916 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074918 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074921 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074923 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074926 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074928 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074930 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074933 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074935 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074938 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074940 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074943 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.085327 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074945 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074948 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074951 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074953 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074956 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074959 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074961 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074963 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074966 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074968 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074971 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074974 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074976 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074979 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074982 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074984 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074987 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074989 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074992 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.085904 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074995 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074997 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.074999 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075002 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075004 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075007 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075009 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075012 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075015 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075017 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075021 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075024 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075027 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075030 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075032 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075035 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.086497 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075038 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.075062 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075095 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075099 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075101 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075104 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075107 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075109 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075111 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075114 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075117 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075120 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075122 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075125 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075127 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075133 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:22:20.086932 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075136 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075138 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075140 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075143 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075145 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075151 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075154 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075156 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075159 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075161 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075164 2578 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075167 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075169 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075172 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075176 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075178 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075181 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075183 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075186 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075189 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:22:20.087373 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075191 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075194 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075196 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075199 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075201 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075204 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075206 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075209 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075211 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075213 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075216 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075219 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075223 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075227 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075230 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075233 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075235 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075239 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075242 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:22:20.087860 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075245 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075247 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075250 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075252 2578 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075255 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075258 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075260 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075263 2578 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075266 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075268 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075271 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075273 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075276 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075278 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075281 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075283 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075285 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075288 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075290 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:22:20.088427 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075293 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075295 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075298 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075300 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075303 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075305 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075308 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075311 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075314 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075317 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075319 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075323 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075325 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075327 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075330 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075332 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075335 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075337 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075340 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:22:20.088992 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075342 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075346 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075348 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075351 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075353 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075356 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075358 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075361 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075365 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075368 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075371 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075373 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075376 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075378 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:20.075380 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.075441 2578 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.078608 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.079413 2578 server.go:1014] "Starting client certificate rotation" Nov 02 17:22:20.089494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.079521 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:22:20.090051 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.079570 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:22:20.090051 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.081138 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:22:20.090051 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.084590 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:22:20.095050 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.095027 2578 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:22:20.097978 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.097959 2578 log.go:25] "Validated CRI v1 image API" Nov 02 17:22:20.098830 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.098804 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:22:20.101293 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.101264 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ada1dd5-3247-49ef-91de-acec385de42d:/dev/nvme0n1p4 bc8d1faa-9c72-4dd2-bd08-a619e9972794:/dev/nvme0n1p3] Nov 02 17:22:20.101336 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.101296 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 17:22:20.107291 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.107170 2578 manager.go:217] Machine: {Timestamp:2026-11-02 17:22:20.105698898 +0000 UTC m=+0.281900467 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107257 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e6e8a1fcdbf9060b2408b70efa9e7 SystemUUID:ec2e6e8a-1fcd-bf90-60b2-408b70efa9e7 BootID:1b55da2b-d252-4a19-acf6-7dabd7f06c77 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:97:a2:5a:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:97:a2:5a:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:b0:dd:7b:2d:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:22:20.107710 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.107700 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:22:20.107860 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.107846 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:22:20.110277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.110246 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:22:20.110422 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.110279 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-71.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:22:20.110468 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.110424 2578 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:22:20.110468 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.110433 2578 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:22:20.110468 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.110458 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:22:20.110543 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.110472 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:22:20.111152 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.111142 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:22:20.114668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.114656 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:22:20.114799 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.114789 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:22:20.115337 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.115328 2578 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:22:20.115366 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.115342 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:22:20.115366 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.115352 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:22:20.115366 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.115363 2578 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:22:20.115460 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.115371 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:22:20.115771 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.115757 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:22:20.117851 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.117823 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:22:20.122965 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.122947 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:22:20.123893 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.123874 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:22:20.123965 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.123910 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:22:20.124143 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124133 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:22:20.124200 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124149 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:22:20.124200 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124155 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:22:20.124200 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124161 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:22:20.124200 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124167 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:22:20.124200 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124173 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:22:20.124567 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124558 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:22:20.124600 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124568 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:22:20.124600 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124576 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:22:20.124600 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124582 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:22:20.124600 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124590 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:22:20.124600 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.124599 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:22:20.125116 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.125106 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:22:20.125116 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.125116 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:22:20.128770 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.128755 2578 server.go:1263] "Started kubelet" Nov 02 17:22:20.128886 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.128829 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:22:20.128977 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.128911 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:22:20.129077 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.128998 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:22:20.129632 ip-10-0-128-71 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:22:20.129954 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.129939 2578 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:22:20.130799 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.130783 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:22:20.134231 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.134208 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 17:22:20.134328 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.134315 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:22:20.134394 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.134332 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:22:20.134528 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.134498 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" Nov 02 17:22:20.134886 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.134866 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:22:20.135164 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.135146 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:22:20.135164 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.135164 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:22:20.135463 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.135289 2578 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:22:20.135463 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.135299 2578 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:22:20.136947 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.136934 2578 factory.go:153] Registering CRI-O factory Nov 02 17:22:20.137020 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.136965 2578 factory.go:223] Registration of the crio container factory successfully Nov 02 17:22:20.137020 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.137013 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:22:20.137020 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.137019 2578 factory.go:55] Registering systemd factory Nov 02 17:22:20.137108 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.137026 2578 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:22:20.137108 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.137044 2578 factory.go:103] Registering Raw factory Nov 02 17:22:20.137108 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.137055 2578 manager.go:1199] Started watching for new ooms in manager Nov 02 17:22:20.138320 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.138301 2578 manager.go:319] Starting recovery of all containers Nov 02 17:22:20.149033 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.149014 2578 manager.go:324] Recovery completed Nov 02 17:22:20.151242 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.151209 2578 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 17:22:20.154712 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.154700 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.156443 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.156420 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.156493 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.156459 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.156493 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.156473 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.157081 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157067 2578 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:22:20.157143 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157081 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:22:20.157143 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157067 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.157143 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157102 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:22:20.157836 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157823 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.157876 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157852 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.157876 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.157865 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.159304 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.159288 2578 policy_none.go:50] "Start" Nov 02 17:22:20.159304 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.159305 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:22:20.159396 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.159315 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:22:20.161273 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.161262 2578 policy_none.go:44] "Start" Nov 02 17:22:20.197456 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.197309 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:22:20.201385 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201368 2578 manager.go:341] "Starting Device Plugin manager" Nov 02 17:22:20.201482 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.201396 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:22:20.201482 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201405 2578 server.go:87] "Starting device plugin registration server" Nov 02 17:22:20.201627 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201615 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:22:20.201698 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201626 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:22:20.201779 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201727 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:22:20.201874 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201851 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:22:20.201874 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.201863 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:22:20.202545 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.202506 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:22:20.202640 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.202590 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" Nov 02 17:22:20.269912 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.269879 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:22:20.272023 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.271250 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:22:20.272023 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.271268 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:22:20.272023 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.271287 2578 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:22:20.272023 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.271321 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 17:22:20.273174 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.273146 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-71.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 17:22:20.274005 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.273147 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-71.us-west-1.compute.internal.18e44a344f40d22b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-71.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-71.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 17:22:20.128703019 +0000 UTC m=+0.304904588,LastTimestamp:2026-11-02 17:22:20.128703019 +0000 UTC m=+0.304904588,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-71.us-west-1.compute.internal,}" Nov 02 17:22:20.301755 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.301715 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.302695 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.302678 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.302818 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.302714 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.302818 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.302747 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.302818 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.302789 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.321826 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.321752 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-71.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 17:22:20.352241 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.352219 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:22:20.352332 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.352288 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56dts" Nov 02 17:22:20.372101 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.372073 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal"] Nov 02 17:22:20.372218 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.372149 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.374439 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.374419 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.374546 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.374448 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.374546 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.374471 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.376894 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.376880 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.377067 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.377133 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377092 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.377848 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377832 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.377951 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377866 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.377951 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377883 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.377951 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377868 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.378215 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377977 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.378215 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.377990 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.380106 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.380088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.380192 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.380122 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:22:20.384519 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.384500 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:22:20.384601 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.384534 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:22:20.384601 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.384553 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:22:20.399336 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.399308 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.403397 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.403377 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.429426 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.429402 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:20.430897 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.430875 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:20.528186 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.528156 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:20.531391 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.531334 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.531548 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:20.531414 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-71.us-west-1.compute.internal\": node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" Nov 02 17:22:20.532875 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.532848 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:20.534890 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.534867 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.536149 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.536129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.536234 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.536199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.536234 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.536222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8894bf200d7f2fbf8f4d4549108d68f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a8894bf200d7f2fbf8f4d4549108d68f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.596003 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.595978 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:22:20.596108 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.596084 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.636919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.636889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.636993 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.636926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.636993 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.636950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8894bf200d7f2fbf8f4d4549108d68f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a8894bf200d7f2fbf8f4d4549108d68f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.637090 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.637004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8894bf200d7f2fbf8f4d4549108d68f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a8894bf200d7f2fbf8f4d4549108d68f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.637090 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.637008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.637090 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.637028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.652966 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.652944 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:22:20.703032 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.703005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:20.706138 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:20.706113 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:22:21.079260 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.079184 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 17:22:21.079834 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.079319 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:22:21.079834 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.079348 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:22:21.079834 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.079382 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 17:22:21.079834 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.079391 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://acf688e646e8844ffb176668e16206eb-5f8c15bb6010e1e3.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-128-71.us-west-1.compute.internal.18e44a3450e81d7f\": read tcp 10.0.128.71:40818->18.220.170.94:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-128-71.us-west-1.compute.internal.18e44a3450e81d7f default 5086 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-71.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-71.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-71.us-west-1.compute.internal,},FirstTimestamp:2026-11-02 17:22:20 +0000 UTC,LastTimestamp:2026-11-02 17:22:20.37443419 +0000 UTC m=+0.550635758,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-71.us-west-1.compute.internal,}" Nov 02 17:22:21.116141 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.116115 2578 apiserver.go:52] "Watching apiserver" Nov 02 17:22:21.135953 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.135925 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 17:22:21.356456 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.356419 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:17:20 +0000 UTC" deadline="2028-06-14 01:20:27.49400625 +0000 UTC" Nov 02 17:22:21.356456 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.356453 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14143h58m6.137558266s" Nov 02 17:22:21.382490 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.382466 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:22:21.382784 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.382767 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-dj5gx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal","openshift-network-diagnostics/network-check-target-tlgwq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p","openshift-multus/multus-additional-cni-plugins-dc65z","openshift-multus/multus-bwsqh","openshift-ovn-kubernetes/ovnkube-node-67tdm","kube-system/global-pull-secret-syncer-pxq56","openshift-image-registry/node-ca-w8qzs","openshift-network-operator/iptables-alerter-hq2f5","openshift-dns/node-resolver-2l4bx","openshift-multus/network-metrics-daemon-hh9rw","kube-system/konnectivity-agent-x89xb"] Nov 02 17:22:21.385343 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.385309 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.387314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.387292 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.389263 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.389247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.391245 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.391227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:21.391304 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.391288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:21.393250 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.393236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.395358 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.395345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.397423 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.397407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:21.397492 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.397479 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:21.399563 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.399541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.403887 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.403865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.406110 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.406091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.408218 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.408205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.408280 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.408267 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:21.410370 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.410355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.436796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.436778 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:22:21.437465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.437451 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:22:21.437518 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.437491 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.437518 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.437511 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kdpj\"" Nov 02 17:22:21.437595 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.437546 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.438459 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.438442 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.438533 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.438507 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.438533 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.438529 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkkxw\"" Nov 02 17:22:21.440148 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.440131 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:22:21.440184 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.440172 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:22:21.440300 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.440290 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-867zk\"" Nov 02 17:22:21.442446 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442428 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-multus\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.442493 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-kubelet\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.442493 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-conf\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.442493 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.442632 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.442685 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-kubernetes\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.442685 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-system-cni-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.442801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442699 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-daemon-config\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.442801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442745 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-tmp\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.442801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-registration-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.442923 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442803 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf40ad1-2c0b-4b84-9242-4108e40de77f-host-slash\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.442923 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-modprobe-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.442923 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-sys-fs\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.442923 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949dj\" (UniqueName: \"kubernetes.io/projected/d8a199fd-2c80-44c0-9930-19820009ada9-kube-api-access-949dj\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.442923 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrcx\" (UniqueName: \"kubernetes.io/projected/b26856a8-af0c-48c4-b6a6-8042e9bea094-kube-api-access-tjrcx\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cni-binary-copy\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.442974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-conf-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-systemd-units\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-script-lib\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443103 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed54c86c-1503-4c0d-b0b7-0454818c844c-tmp-dir\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.443126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-netns\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgf5\" (UniqueName: \"kubernetes.io/projected/9cf40ad1-2c0b-4b84-9242-4108e40de77f-kube-api-access-cqgf5\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-etc-tuned\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443262 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45f8e074-1aca-444b-91e3-5f5bc339164f-agent-certs\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed54c86c-1503-4c0d-b0b7-0454818c844c-hosts-file\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qfg\" (UniqueName: \"kubernetes.io/projected/0c38f276-7bda-4a9d-a2b9-3e07385ed119-kube-api-access-k5qfg\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-kubelet\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-slash\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443407 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovn-node-metrics-cert\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-multus-certs\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf40ad1-2c0b-4b84-9242-4108e40de77f-iptables-alerter-script\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976gb\" (UniqueName: \"kubernetes.io/projected/bcd31aff-608d-4200-b76d-61f4c43e8a99-kube-api-access-976gb\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443486 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-bin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-sys\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-host\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-socket-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443618 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-systemd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-bin\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443704 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stl27\" (UniqueName: \"kubernetes.io/projected/ed54c86c-1503-4c0d-b0b7-0454818c844c-kube-api-access-stl27\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443837 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-hostroot\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.443850 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-lib-modules\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-env-overrides\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzmp\" (UniqueName: \"kubernetes.io/projected/63987b50-f8b5-42a2-a5cf-50fc6cec635e-kube-api-access-rxzmp\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-socket-dir-parent\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443960 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-netns\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.443990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-etc-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444017 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b26856a8-af0c-48c4-b6a6-8042e9bea094-serviceca\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444035 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-system-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45f8e074-1aca-444b-91e3-5f5bc339164f-konnectivity-ca\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cnibin\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444114 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2nz\" (UniqueName: \"kubernetes.io/projected/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-kube-api-access-lq2nz\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-var-lib-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-ovn\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgsw\" (UniqueName: \"kubernetes.io/projected/e6541049-99e9-453c-9953-8817f491dc4d-kube-api-access-khgsw\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-kubelet-config\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.444424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444228 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444261 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-log-socket\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444279 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-systemd\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-run\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444409 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444421 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444435 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-node-log\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-config\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-dbus\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-device-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysconfig\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cnibin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-k8s-cni-cncf-io\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-var-lib-kubelet\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-etc-kubernetes\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.444919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-netd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.445404 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b26856a8-af0c-48c4-b6a6-8042e9bea094-host\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.445404 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.444806 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4464\"" Nov 02 17:22:21.446240 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.446227 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.534113 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.534086 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 17:22:21.534230 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.534135 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w7rxv" Nov 02 17:22:21.537600 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537584 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.537770 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537728 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:22:21.537894 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537782 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:22:21.537894 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537803 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.537894 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537831 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:22:21.538066 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537905 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.538066 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537922 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.538066 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.537955 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:22:21.538066 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538014 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.538066 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538055 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:22:21.538066 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538061 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:22:21.538325 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538113 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:22:21.538325 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538137 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:22:21.538325 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538150 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:22:21.538325 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538169 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:22:21.538325 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538193 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wf5x\"" Nov 02 17:22:21.538459 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538345 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztdmt\"" Nov 02 17:22:21.538459 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.538393 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-546w9\"" Nov 02 17:22:21.544500 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.544486 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2rxq\"" Nov 02 17:22:21.544555 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.544516 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jxqfl\"" Nov 02 17:22:21.544601 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.544591 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:22:21.545646 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:21.545720 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.545720 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-hostroot\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.545840 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545726 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-lib-modules\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.545840 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545759 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-hostroot\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.545840 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.545840 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-env-overrides\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.545840 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545803 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.545840 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzmp\" (UniqueName: \"kubernetes.io/projected/63987b50-f8b5-42a2-a5cf-50fc6cec635e-kube-api-access-rxzmp\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545852 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-socket-dir-parent\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-netns\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-etc-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545925 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b26856a8-af0c-48c4-b6a6-8042e9bea094-serviceca\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545936 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-socket-dir-parent\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-system-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545958 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-lib-modules\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545967 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-etc-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545972 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45f8e074-1aca-444b-91e3-5f5bc339164f-konnectivity-ca\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-netns\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.545996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cnibin\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546024 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2nz\" (UniqueName: \"kubernetes.io/projected/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-kube-api-access-lq2nz\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-system-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546034 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cnibin\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-var-lib-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-ovn\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546122 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546075 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgsw\" (UniqueName: \"kubernetes.io/projected/e6541049-99e9-453c-9953-8817f491dc4d-kube-api-access-khgsw\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-kubelet-config\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-var-lib-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-log-socket\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-systemd\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-run\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-node-log\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-config\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546298 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-systemd\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-dbus\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-device-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546342 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-log-socket\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysconfig\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cnibin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.546933 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-k8s-cni-cncf-io\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546435 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b26856a8-af0c-48c4-b6a6-8042e9bea094-serviceca\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-env-overrides\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546480 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-var-lib-kubelet\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546521 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-run\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-var-lib-kubelet\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-etc-kubernetes\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546575 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-netd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45f8e074-1aca-444b-91e3-5f5bc339164f-konnectivity-ca\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b26856a8-af0c-48c4-b6a6-8042e9bea094-host\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546626 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-netd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-multus\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-dbus\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546685 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-kubelet\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-conf\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysconfig\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.547617 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-ovn\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cnibin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-kubernetes\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546838 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-k8s-cni-cncf-io\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546842 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-system-cni-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546876 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-multus\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-daemon-config\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-tmp\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546927 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-device-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-registration-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf40ad1-2c0b-4b84-9242-4108e40de77f-host-slash\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-modprobe-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547013 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b26856a8-af0c-48c4-b6a6-8042e9bea094-host\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547026 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-conf\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.548506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-kubelet\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.546576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-node-log\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-sys-fs\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547057 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-sys-fs\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-registration-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949dj\" (UniqueName: \"kubernetes.io/projected/d8a199fd-2c80-44c0-9930-19820009ada9-kube-api-access-949dj\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrcx\" (UniqueName: \"kubernetes.io/projected/b26856a8-af0c-48c4-b6a6-8042e9bea094-kube-api-access-tjrcx\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.547121 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf40ad1-2c0b-4b84-9242-4108e40de77f-host-slash\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cni-binary-copy\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-etc-kubernetes\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-conf-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.547188 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:22:22.047161988 +0000 UTC m=+2.223363576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547202 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-modprobe-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547203 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-conf-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.549323 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547205 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-systemd-units\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547270 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-script-lib\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed54c86c-1503-4c0d-b0b7-0454818c844c-tmp-dir\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-netns\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgf5\" (UniqueName: \"kubernetes.io/projected/9cf40ad1-2c0b-4b84-9242-4108e40de77f-kube-api-access-cqgf5\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547354 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-system-cni-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-etc-tuned\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45f8e074-1aca-444b-91e3-5f5bc339164f-agent-certs\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547394 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-kubernetes\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed54c86c-1503-4c0d-b0b7-0454818c844c-hosts-file\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547423 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547429 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550136 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qfg\" (UniqueName: \"kubernetes.io/projected/0c38f276-7bda-4a9d-a2b9-3e07385ed119-kube-api-access-k5qfg\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-kubelet\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-systemd-units\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed54c86c-1503-4c0d-b0b7-0454818c844c-hosts-file\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547512 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-config\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547424 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-netns\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547801 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-kubelet\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed54c86c-1503-4c0d-b0b7-0454818c844c-tmp-dir\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.547884 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.547939 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:22.047923588 +0000 UTC m=+2.224125157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cni-binary-copy\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.547988 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-kubelet-config\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-script-lib\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-slash\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548131 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovn-node-metrics-cert\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-multus-certs\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.550823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf40ad1-2c0b-4b84-9242-4108e40de77f-iptables-alerter-script\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-multus-certs\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548210 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976gb\" (UniqueName: \"kubernetes.io/projected/bcd31aff-608d-4200-b76d-61f4c43e8a99-kube-api-access-976gb\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-bin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548262 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-sys\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-host\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-socket-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-systemd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-bin\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stl27\" (UniqueName: \"kubernetes.io/projected/ed54c86c-1503-4c0d-b0b7-0454818c844c-kube-api-access-stl27\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548182 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-slash\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548445 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-host\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-bin\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548557 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-systemd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548592 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-bin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-sys\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.548994 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-socket-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.551277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.550075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.550162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.550221 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-daemon-config\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.550463 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf40ad1-2c0b-4b84-9242-4108e40de77f-iptables-alerter-script\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.550623 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-etc-tuned\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.550687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-tmp\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.551016 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45f8e074-1aca-444b-91e3-5f5bc339164f-agent-certs\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.551796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.551770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovn-node-metrics-cert\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.600593 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.600550 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:21.600593 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.600574 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:21.600593 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.600589 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:21.600888 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:21.600691 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:22.100667857 +0000 UTC m=+2.276869428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:21.603278 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.603248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2nz\" (UniqueName: \"kubernetes.io/projected/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-kube-api-access-lq2nz\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.603411 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.603341 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzmp\" (UniqueName: \"kubernetes.io/projected/63987b50-f8b5-42a2-a5cf-50fc6cec635e-kube-api-access-rxzmp\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.603681 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.603659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrcx\" (UniqueName: \"kubernetes.io/projected/b26856a8-af0c-48c4-b6a6-8042e9bea094-kube-api-access-tjrcx\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.603942 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.603925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgsw\" (UniqueName: \"kubernetes.io/projected/e6541049-99e9-453c-9953-8817f491dc4d-kube-api-access-khgsw\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.604164 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.604142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qfg\" (UniqueName: \"kubernetes.io/projected/0c38f276-7bda-4a9d-a2b9-3e07385ed119-kube-api-access-k5qfg\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.604640 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.604621 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgf5\" (UniqueName: \"kubernetes.io/projected/9cf40ad1-2c0b-4b84-9242-4108e40de77f-kube-api-access-cqgf5\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.604906 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.604890 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-949dj\" (UniqueName: \"kubernetes.io/projected/d8a199fd-2c80-44c0-9930-19820009ada9-kube-api-access-949dj\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.607749 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.607676 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976gb\" (UniqueName: \"kubernetes.io/projected/bcd31aff-608d-4200-b76d-61f4c43e8a99-kube-api-access-976gb\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:21.609220 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.609200 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stl27\" (UniqueName: \"kubernetes.io/projected/ed54c86c-1503-4c0d-b0b7-0454818c844c-kube-api-access-stl27\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.716745 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.716695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:22:21.736825 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.736796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:22:21.755585 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.755539 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:21.775844 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.775804 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:22:21.801513 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.801479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:22:21.803766 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.803728 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwsqh" Nov 02 17:22:21.820368 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.820340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:22:21.838452 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.838427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:22:21.860925 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.860903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:21.870386 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:21.870349 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf40ad1_2c0b_4b84_9242_4108e40de77f.slice/crio-e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38 WatchSource:0}: Error finding container e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38: Status 404 returned error can't find the container with id e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38 Nov 02 17:22:21.871264 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:21.871240 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8894bf200d7f2fbf8f4d4549108d68f.slice/crio-052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8 WatchSource:0}: Error finding container 052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8: Status 404 returned error can't find the container with id 052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8 Nov 02 17:22:21.872107 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:21.872082 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94799fcc07ccf47af40bea71d377682.slice/crio-1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f WatchSource:0}: Error finding container 1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f: Status 404 returned error can't find the container with id 1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f Nov 02 17:22:21.876048 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:21.876030 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:22:21.877302 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:21.877278 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63987b50_f8b5_42a2_a5cf_50fc6cec635e.slice/crio-19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff WatchSource:0}: Error finding container 19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff: Status 404 returned error can't find the container with id 19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff Nov 02 17:22:22.052770 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.052729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:22.052974 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.052820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:22.052974 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.052909 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:22.052974 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.052908 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:22.053070 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.052979 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:22:23.052965281 +0000 UTC m=+3.229166842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:22.053070 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.052994 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:23.052987913 +0000 UTC m=+3.229189471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:22.068607 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.068582 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:22.153475 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.153375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:22.154102 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.153557 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:22.154102 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.153577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:22.154102 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.153587 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:22.154102 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:22.153646 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:23.153627573 +0000 UTC m=+3.329829143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:22.276014 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.275962 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff"} Nov 02 17:22:22.277031 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.277007 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hq2f5" event={"ID":"9cf40ad1-2c0b-4b84-9242-4108e40de77f","Type":"ContainerStarted","Data":"e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38"} Nov 02 17:22:22.278126 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.278105 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a94799fcc07ccf47af40bea71d377682","Type":"ContainerStarted","Data":"1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f"} Nov 02 17:22:22.279162 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.279143 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a8894bf200d7f2fbf8f4d4549108d68f","Type":"ContainerStarted","Data":"052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8"} Nov 02 17:22:22.482184 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.482103 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:22.521700 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.521675 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:22:22.537447 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.537392 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:21 +0000 UTC" deadline="2028-07-06 14:17:06.867016884 +0000 UTC" Nov 02 17:22:22.537447 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:22.537443 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14684h54m44.329577027s" Nov 02 17:22:22.578300 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:22.578271 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded54c86c_1503_4c0d_b0b7_0454818c844c.slice/crio-0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb WatchSource:0}: Error finding container 0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb: Status 404 returned error can't find the container with id 0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb Nov 02 17:22:22.657396 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:22.657366 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f458e79_ca1c_4eb2_9ddd_f3daacf6055a.slice/crio-2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f WatchSource:0}: Error finding container 2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f: Status 404 returned error can't find the container with id 2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f Nov 02 17:22:22.697264 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:22.697230 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c38f276_7bda_4a9d_a2b9_3e07385ed119.slice/crio-11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e WatchSource:0}: Error finding container 11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e: Status 404 returned error can't find the container with id 11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e Nov 02 17:22:22.698614 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:22.698588 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a199fd_2c80_44c0_9930_19820009ada9.slice/crio-7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f WatchSource:0}: Error finding container 7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f: Status 404 returned error can't find the container with id 7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f Nov 02 17:22:22.702945 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:22:22.702918 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f8e074_1aca_444b_91e3_5f5bc339164f.slice/crio-2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6 WatchSource:0}: Error finding container 2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6: Status 404 returned error can't find the container with id 2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6 Nov 02 17:22:23.059702 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.059612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:23.059702 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.059682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:23.059991 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.059818 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:23.059991 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.059872 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:22:25.059858916 +0000 UTC m=+5.236060477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:23.059991 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.059928 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:23.059991 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.059957 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:25.059945335 +0000 UTC m=+5.236146895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:23.161062 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.160996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:23.161529 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.161241 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:23.161529 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.161266 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:23.161529 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.161280 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:23.161529 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.161350 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:25.161329299 +0000 UTC m=+5.337530863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:23.272395 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.272362 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:23.272588 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.272494 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:23.272588 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.272574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:23.272705 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.272627 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:23.272705 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.272683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:23.272837 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:23.272739 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:23.283867 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.283832 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2l4bx" event={"ID":"ed54c86c-1503-4c0d-b0b7-0454818c844c","Type":"ContainerStarted","Data":"0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb"} Nov 02 17:22:23.287270 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.287238 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" event={"ID":"e6541049-99e9-453c-9953-8817f491dc4d","Type":"ContainerStarted","Data":"82a7d6c87c5cb903940c0b8cbdd30785e8ae8f8a053fff54b33a448a3436d351"} Nov 02 17:22:23.291285 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.291257 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x89xb" event={"ID":"45f8e074-1aca-444b-91e3-5f5bc339164f","Type":"ContainerStarted","Data":"2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6"} Nov 02 17:22:23.303524 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.303488 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" event={"ID":"d8a199fd-2c80-44c0-9930-19820009ada9","Type":"ContainerStarted","Data":"7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f"} Nov 02 17:22:23.305413 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.305389 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwsqh" event={"ID":"0c38f276-7bda-4a9d-a2b9-3e07385ed119","Type":"ContainerStarted","Data":"11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e"} Nov 02 17:22:23.307826 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.307796 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerStarted","Data":"2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f"} Nov 02 17:22:23.311164 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.311091 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8qzs" event={"ID":"b26856a8-af0c-48c4-b6a6-8042e9bea094","Type":"ContainerStarted","Data":"c13dd52c17fff47bebf980e36af54c6cae71de5eb4acdbb0997f8b93bb343668"} Nov 02 17:22:23.537895 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.537826 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:21 +0000 UTC" deadline="2028-07-28 04:23:46.99797218 +0000 UTC" Nov 02 17:22:23.537895 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:23.537892 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15203h1m23.460084844s" Nov 02 17:22:25.077246 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:25.077205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:25.077796 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:25.077298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:25.077796 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.077449 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:25.077796 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.077517 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:22:29.077497278 +0000 UTC m=+9.253698839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:25.077796 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.077587 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:25.077796 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.077622 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:29.077609609 +0000 UTC m=+9.253811169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:25.178770 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:25.178723 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:25.179004 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.178952 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:25.179004 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.178976 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:25.179004 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.178993 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:25.179197 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.179058 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:29.179037925 +0000 UTC m=+9.355239493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:25.271684 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:25.271646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:25.271922 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.271812 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:25.271922 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:25.271890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:25.272057 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.271974 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:25.272057 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:25.272033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:25.272197 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:25.272122 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:27.272072 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.271982 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:27.272560 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:27.272127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:27.272560 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.272204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:27.272560 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:27.272285 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:27.272560 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.272342 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:27.272560 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:27.272432 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:27.336894 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.336822 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a8894bf200d7f2fbf8f4d4549108d68f","Type":"ContainerStarted","Data":"8ed51046e461248943a1f736a48cad9b9e86e78d10afc13db52a38a37ae80a2c"} Nov 02 17:22:27.341002 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.340966 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="a9e9c513df1a3b12779a4ccb89a0835b8f72eb823ec57d07fa9d5b2bd516e644" exitCode=0 Nov 02 17:22:27.341439 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.341371 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerDied","Data":"a9e9c513df1a3b12779a4ccb89a0835b8f72eb823ec57d07fa9d5b2bd516e644"} Nov 02 17:22:27.348755 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.348720 2578 generic.go:356] "Generic (PLEG): container finished" podID="a94799fcc07ccf47af40bea71d377682" containerID="d7743a49a497c925f3daed0eb78afa202f453aa245a4ed622af3801f6500928e" exitCode=0 Nov 02 17:22:27.348897 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.348777 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a94799fcc07ccf47af40bea71d377682","Type":"ContainerDied","Data":"d7743a49a497c925f3daed0eb78afa202f453aa245a4ed622af3801f6500928e"} Nov 02 17:22:27.462461 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:27.462400 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" podStartSLOduration=6.462384563 podStartE2EDuration="6.462384563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:27.461664136 +0000 UTC m=+7.637865734" watchObservedRunningTime="2026-11-02 17:22:27.462384563 +0000 UTC m=+7.638586144" Nov 02 17:22:28.357596 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:28.357534 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x89xb" event={"ID":"45f8e074-1aca-444b-91e3-5f5bc339164f","Type":"ContainerStarted","Data":"27bc34e2b19fc933cedf200b82a24f692c23464d6e18f4075314e6dbb386a99a"} Nov 02 17:22:28.370074 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:28.370033 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a94799fcc07ccf47af40bea71d377682","Type":"ContainerStarted","Data":"f2a2af8c18445ef5bec69cc7c7e8677e5ff2a1a8f168995e53e6954680c52c93"} Nov 02 17:22:28.597487 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:28.597418 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x89xb" podStartSLOduration=3.380723767 podStartE2EDuration="8.597394831s" totalImagesPullingTime="5.216671064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:28.478247072 +0000 UTC m=+8.654448660" watchObservedRunningTime="2026-11-02 17:22:28.597394831 +0000 UTC m=+8.773596422" Nov 02 17:22:28.597736 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:28.597695 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" podStartSLOduration=7.597684716 podStartE2EDuration="7.597684716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:28.597160004 +0000 UTC m=+8.773361610" watchObservedRunningTime="2026-11-02 17:22:28.597684716 +0000 UTC m=+8.773886311" Nov 02 17:22:29.114581 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.114544 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:29.114716 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.114606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:29.114786 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.114772 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:29.114870 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.114827 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:37.114809062 +0000 UTC m=+17.291010632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:29.114924 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.114891 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:29.114956 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.114925 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:22:37.114913276 +0000 UTC m=+17.291114845 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:29.215305 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.214939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:29.215305 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.215169 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:29.215305 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.215191 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:29.215305 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.215215 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:29.215305 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.215276 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:37.215257491 +0000 UTC m=+17.391459048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:29.272327 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.271847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:29.272327 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.272000 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:29.272327 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.272075 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:29.272327 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.272153 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:29.272327 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.272210 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:29.272327 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:29.272280 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:29.377433 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.377355 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w8qzs" event={"ID":"b26856a8-af0c-48c4-b6a6-8042e9bea094","Type":"ContainerStarted","Data":"ae2397a91c3ccdcac10e16dfc130725a91f7395826f63ba990ace0231dd4cae1"} Nov 02 17:22:29.514159 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.514127 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:29.514746 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.514706 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:29.624428 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:29.624374 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w8qzs" podStartSLOduration=3.253445756 podStartE2EDuration="9.624354257s" totalImagesPullingTime="6.370908501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:29.486583141 +0000 UTC m=+9.662784723" watchObservedRunningTime="2026-11-02 17:22:29.624354257 +0000 UTC m=+9.800555836" Nov 02 17:22:30.381812 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:30.381773 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" event={"ID":"d8a199fd-2c80-44c0-9930-19820009ada9","Type":"ContainerStarted","Data":"9406e8b21913000b39e9426258b200b80d2441a31b7678b1126a30c1f02d0a66"} Nov 02 17:22:30.382206 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:30.381850 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:30.383314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:30.383286 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:22:31.272073 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:31.272047 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:31.272267 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:31.272141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:31.272267 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:31.272202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:31.272267 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:31.272247 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:31.272404 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:31.272285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:31.272404 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:31.272333 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:32.130484 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.130331 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:22:32.215973 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.215876 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:22:32.13047719Z","UUID":"7a21bc90-542f-4195-983f-969b42e824b7","Handler":null,"Name":"","Endpoint":""} Nov 02 17:22:32.217698 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.217674 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:22:32.217849 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.217709 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:22:32.387950 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.387913 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerStarted","Data":"a57276a1bf7264356604c9d47cefb7899645cb6417c7a9dd59cdbd76219276a6"} Nov 02 17:22:32.390381 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.390349 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2l4bx" event={"ID":"ed54c86c-1503-4c0d-b0b7-0454818c844c","Type":"ContainerStarted","Data":"a8092ffc038e098061ddc9ac2883a9ea8bb64c27b6af313d2a9a2202a817d450"} Nov 02 17:22:32.392714 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.392689 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" event={"ID":"d8a199fd-2c80-44c0-9930-19820009ada9","Type":"ContainerStarted","Data":"5a035a85a0c18be9510439fa7c04ef75cbeee77e4820a62ede9008cceff5b75e"} Nov 02 17:22:32.395089 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.395064 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwsqh" event={"ID":"0c38f276-7bda-4a9d-a2b9-3e07385ed119","Type":"ContainerStarted","Data":"c8c90f3c6178c73d920b60b85111262592952b9b04b3585022bddacf2935ef77"} Nov 02 17:22:32.732120 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.732066 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-2l4bx" podStartSLOduration=3.7426795889999998 podStartE2EDuration="12.732050816s" totalImagesPullingTime="8.989371227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:32.618555705 +0000 UTC m=+12.794757284" watchObservedRunningTime="2026-11-02 17:22:32.732050816 +0000 UTC m=+12.908252439" Nov 02 17:22:32.732614 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:32.732568 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-bwsqh" podStartSLOduration=2.875413685 podStartE2EDuration="11.732554247s" totalImagesPullingTime="8.857140562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:32.731572935 +0000 UTC m=+12.907774515" watchObservedRunningTime="2026-11-02 17:22:32.732554247 +0000 UTC m=+12.908755827" Nov 02 17:22:33.271746 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.271700 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:33.272351 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:33.271843 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:33.272351 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.271871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:33.272351 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.271878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:33.272351 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:33.272000 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:33.272351 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:33.272038 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:33.400280 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.400239 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hq2f5" event={"ID":"9cf40ad1-2c0b-4b84-9242-4108e40de77f","Type":"ContainerStarted","Data":"8c1e56d9d60f1f989a9ecb3055fb4ab876bf76a5076aa79215515c23ac2f5168"} Nov 02 17:22:33.403158 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.403122 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="a57276a1bf7264356604c9d47cefb7899645cb6417c7a9dd59cdbd76219276a6" exitCode=0 Nov 02 17:22:33.404198 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.404174 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerDied","Data":"a57276a1bf7264356604c9d47cefb7899645cb6417c7a9dd59cdbd76219276a6"} Nov 02 17:22:33.511256 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:33.511192 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hq2f5" podStartSLOduration=3.610649387 podStartE2EDuration="12.511172571s" totalImagesPullingTime="8.900523184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:33.510057449 +0000 UTC m=+13.686259026" watchObservedRunningTime="2026-11-02 17:22:33.511172571 +0000 UTC m=+13.687374148" Nov 02 17:22:34.406869 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:34.406817 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" event={"ID":"e6541049-99e9-453c-9953-8817f491dc4d","Type":"ContainerStarted","Data":"81836021ed0d41de15e421fb48447872c2592c0019de847ce68dbee83bab9b32"} Nov 02 17:22:34.411709 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:34.410923 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" event={"ID":"d8a199fd-2c80-44c0-9930-19820009ada9","Type":"ContainerStarted","Data":"9fe71b49c22438ca88054e5e3f53a1457ff53056f49e9447857a6af6d7452512"} Nov 02 17:22:34.519428 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:34.519378 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" podStartSLOduration=2.584545395 podStartE2EDuration="13.5193641s" totalImagesPullingTime="10.934818705s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:34.518675257 +0000 UTC m=+14.694876837" watchObservedRunningTime="2026-11-02 17:22:34.5193641 +0000 UTC m=+14.695565680" Nov 02 17:22:34.632404 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:34.632357 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" podStartSLOduration=4.509449472 podStartE2EDuration="14.632343857s" totalImagesPullingTime="10.122894385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:34.63223863 +0000 UTC m=+14.808440208" watchObservedRunningTime="2026-11-02 17:22:34.632343857 +0000 UTC m=+14.808545435" Nov 02 17:22:35.271860 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:35.271816 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:35.272044 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:35.271885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:35.272044 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:35.272005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:35.272044 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:35.272014 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:35.272177 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:35.272105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:35.272177 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:35.272154 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:35.416166 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:35.416123 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="14a40833d54f3c46177c99e584babd8bb9a2f878521278fad73b112f6d286a03" exitCode=0 Nov 02 17:22:35.416671 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:35.416208 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerDied","Data":"14a40833d54f3c46177c99e584babd8bb9a2f878521278fad73b112f6d286a03"} Nov 02 17:22:37.179402 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.179184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:37.179810 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.179437 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:37.179810 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.179338 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:37.179810 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.179537 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:22:53.179515514 +0000 UTC m=+33.355717074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:37.179810 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.179557 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:37.179810 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.179614 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:53.179596201 +0000 UTC m=+33.355797790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:37.272372 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.272177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:37.272372 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.272236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:37.272372 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.272183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:37.272372 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.272324 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:37.272613 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.272412 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:37.272613 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.272488 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:37.280545 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.280512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:37.280722 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.280694 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:37.280843 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.280727 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:37.280843 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.280754 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:37.280843 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:37.280818 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:22:53.28079944 +0000 UTC m=+33.457001019 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:37.422702 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.422669 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="3711ebff2e38eddfc07413aee7f7913d14c6e16541f125aeec6541b7d1807eb3" exitCode=0 Nov 02 17:22:37.422702 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:37.422707 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerDied","Data":"3711ebff2e38eddfc07413aee7f7913d14c6e16541f125aeec6541b7d1807eb3"} Nov 02 17:22:38.428972 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.428791 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:22:38.429402 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429316 2578 generic.go:356] "Generic (PLEG): container finished" podID="63987b50-f8b5-42a2-a5cf-50fc6cec635e" containerID="3d8a40a4f2207b7b616302dfd0b688f873000ddd0a8482c846d8ed5431aae783" exitCode=1 Nov 02 17:22:38.429402 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429356 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"c8ee7c1c13f51005445b8890550e4578ba8c6d64ba598192bc44269a3945dfc6"} Nov 02 17:22:38.429482 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429417 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"dfd8bd3297e13e8f8490bff8e21c2a2a514fd2a341a6711065cadca5d2401bb5"} Nov 02 17:22:38.429482 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429450 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"d01966f25fb0d30438d1d3e19dfb030dabc185906650d6ca06d4eb409490e222"} Nov 02 17:22:38.429546 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429500 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"9ad67d7768ad9ac272573eed011ea50fcf81736c44936eb5b6b35c85fb0a956f"} Nov 02 17:22:38.429546 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429528 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerDied","Data":"3d8a40a4f2207b7b616302dfd0b688f873000ddd0a8482c846d8ed5431aae783"} Nov 02 17:22:38.429609 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:38.429572 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"4fdeb5eacbed159c9163e1751edafdeb9ef94d836f3b2380d02977fc194ab4d7"} Nov 02 17:22:39.272212 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:39.272173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:39.272212 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:39.272205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:39.272452 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:39.272228 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:39.272452 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:39.272350 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:39.272452 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:39.272424 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:39.272564 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:39.272512 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:39.435137 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:39.435101 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerStarted","Data":"735dac0c9905990e3900e28438595fbc95aa4fc8324500b72ff7faca68c46189"} Nov 02 17:22:40.440683 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:40.440655 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:22:40.441475 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:40.441082 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"7eb032c4c65ad0a24b84226ef37c74bda2ecce3acd25750722b000128f085ac6"} Nov 02 17:22:40.443888 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:40.443862 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="735dac0c9905990e3900e28438595fbc95aa4fc8324500b72ff7faca68c46189" exitCode=0 Nov 02 17:22:40.444013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:40.443904 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerDied","Data":"735dac0c9905990e3900e28438595fbc95aa4fc8324500b72ff7faca68c46189"} Nov 02 17:22:41.271963 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:41.271912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:41.271963 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:41.271951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:41.272211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:41.271910 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:41.272211 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:41.272088 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:41.272311 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:41.272268 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:41.272388 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:41.272367 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:41.449351 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:41.449319 2578 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="dd6cc80c8146f12979d66be020d15a7d80c0c7504000cee12f800553aab921fa" exitCode=0 Nov 02 17:22:41.449805 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:41.449366 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerDied","Data":"dd6cc80c8146f12979d66be020d15a7d80c0c7504000cee12f800553aab921fa"} Nov 02 17:22:42.455062 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:42.455022 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dc65z" event={"ID":"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a","Type":"ContainerStarted","Data":"320957c785347d2a0118e46a55cecade85772f430a85fefededae3cb8e3211f9"} Nov 02 17:22:42.662165 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:42.662110 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dc65z" podStartSLOduration=10.280947499 podStartE2EDuration="22.6620933s" totalImagesPullingTime="12.381145801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:42.625367494 +0000 UTC m=+22.801569072" watchObservedRunningTime="2026-11-02 17:22:42.6620933 +0000 UTC m=+22.838294878" Nov 02 17:22:43.272528 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:43.272345 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:43.272678 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:43.272378 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:43.272678 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:43.272418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:43.272785 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:43.272686 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:43.272815 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:43.272776 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:43.272815 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:43.272590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:43.460161 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:43.460134 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:22:43.460613 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:43.460555 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"53075848110c8f5f8022643b8757308cc67e35e5295e31e34d1440c28c84f6a8"} Nov 02 17:22:43.461249 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:43.461228 2578 scope.go:122] "RemoveContainer" containerID="3d8a40a4f2207b7b616302dfd0b688f873000ddd0a8482c846d8ed5431aae783" Nov 02 17:22:44.466651 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.466622 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:22:44.467067 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.466994 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" event={"ID":"63987b50-f8b5-42a2-a5cf-50fc6cec635e","Type":"ContainerStarted","Data":"6abc47f2481edb62691f9b7ab5635465c4395c3740620188404d83f39ea000a4"} Nov 02 17:22:44.467315 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.467287 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:44.467535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.467417 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:44.467535 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.467481 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:44.482624 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.482598 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:44.482762 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.482705 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:22:44.737185 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:44.737084 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" podStartSLOduration=7.866836068 podStartE2EDuration="23.737070189s" totalImagesPullingTime="15.870234121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:22:44.736963653 +0000 UTC m=+24.913165230" watchObservedRunningTime="2026-11-02 17:22:44.737070189 +0000 UTC m=+24.913271790" Nov 02 17:22:45.272216 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.272183 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:45.272368 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.272259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:45.272368 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.272284 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:45.272478 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:45.272371 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:45.272478 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:45.272405 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:45.272538 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:45.272474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:45.284508 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.284472 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tlgwq"] Nov 02 17:22:45.288311 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.288276 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxq56"] Nov 02 17:22:45.288910 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.288890 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hh9rw"] Nov 02 17:22:45.469117 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.469078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:45.469117 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.469097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:45.469117 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:45.469078 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:45.469784 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:45.469214 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:45.469784 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:45.469662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:45.469784 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:45.469746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:47.271794 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:47.271711 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:47.272203 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:47.271756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:47.272203 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:47.271889 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:47.272203 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:47.271764 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:47.272203 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:47.271947 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:47.272203 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:47.272043 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:49.271494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:49.271463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:49.272045 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:49.271610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:49.272045 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:49.271629 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:49.272045 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:49.271648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:49.272045 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:49.271717 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:49.272045 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:49.271833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:50.369131 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:50.368913 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-x89xb_45f8e074-1aca-444b-91e3-5f5bc339164f/konnectivity-agent/0.log" limit=1024 Nov 02 17:22:51.272493 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:51.272457 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:51.272677 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:51.272503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:51.272677 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:51.272563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:51.272677 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:51.272577 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:51.272677 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:51.272625 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:51.272835 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:51.272720 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:53.193489 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:53.193449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:53.193912 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:53.193531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:53.193912 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.193618 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:53.193912 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.193618 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:53.193912 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.193685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret podName:bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e nodeName:}" failed. No retries permitted until 2026-11-02 17:23:25.193666956 +0000 UTC m=+65.369868526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret") pod "global-pull-secret-syncer-pxq56" (UID: "bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e") : object "kube-system"/"original-pull-secret" not registered Nov 02 17:22:53.193912 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.193702 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs podName:bcd31aff-608d-4200-b76d-61f4c43e8a99 nodeName:}" failed. No retries permitted until 2026-11-02 17:23:25.193693876 +0000 UTC m=+65.369895445 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs") pod "network-metrics-daemon-hh9rw" (UID: "bcd31aff-608d-4200-b76d-61f4c43e8a99") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 17:22:53.271697 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:53.271662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:53.271885 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:53.271727 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:53.271885 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:53.271844 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:53.271885 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.271864 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:53.271990 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.271916 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:53.271990 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.271963 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:53.294872 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:53.294843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:53.295019 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.294957 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 17:22:53.295019 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.294969 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 17:22:53.295019 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.294978 2578 projected.go:196] Error preparing data for projected volume kube-api-access-khkqp for pod openshift-network-diagnostics/network-check-target-tlgwq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:53.295019 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:53.295021 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp podName:9021558c-9c68-4634-a371-3172c24a2d85 nodeName:}" failed. No retries permitted until 2026-11-02 17:23:25.295009841 +0000 UTC m=+65.471211398 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-khkqp" (UniqueName: "kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp") pod "network-check-target-tlgwq" (UID: "9021558c-9c68-4634-a371-3172c24a2d85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 17:22:55.264545 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:55.264473 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2l4bx_ed54c86c-1503-4c0d-b0b7-0454818c844c/dns-node-resolver/0.log" Nov 02 17:22:55.271709 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:55.271687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:55.271823 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:55.271799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:55.271823 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:55.271809 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:55.271899 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:55.271836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:55.271936 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:55.271909 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:55.271989 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:55.271974 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:56.662056 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:56.662030 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w8qzs_b26856a8-af0c-48c4-b6a6-8042e9bea094/node-ca/0.log" Nov 02 17:22:57.271992 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:57.271963 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:57.272174 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:57.272055 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:57.272174 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:57.272155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:57.272245 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:57.272161 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:57.272275 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:57.272235 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:22:57.272316 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:57.272299 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:59.272480 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:59.272440 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:22:59.272889 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:59.272489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:22:59.272889 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:22:59.272492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:22:59.272889 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:59.272564 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:22:59.272889 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:59.272639 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:22:59.272889 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:22:59.272723 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:23:01.272447 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:01.272412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:01.272447 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:01.272437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:01.272890 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:01.272444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:01.272890 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:01.272531 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:23:01.272890 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:01.272585 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:23:01.272890 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:01.272633 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:23:03.272028 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:03.271987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:03.272028 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:03.272021 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:03.272450 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:03.272111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:03.272450 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:03.272125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:23:03.272450 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:03.272201 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:23:03.272450 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:03.272262 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:23:05.271570 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:05.271521 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:05.271570 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:05.271574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:05.272222 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:05.271595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:05.272222 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:05.271691 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:23:05.272222 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:05.271814 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:23:05.272222 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:05.271926 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:23:07.272494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.272454 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:07.272494 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.272482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:07.272997 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.272569 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:07.272997 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:07.272580 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tlgwq" podUID="9021558c-9c68-4634-a371-3172c24a2d85" Nov 02 17:23:07.272997 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:07.272701 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hh9rw" podUID="bcd31aff-608d-4200-b76d-61f4c43e8a99" Nov 02 17:23:07.272997 ip-10-0-128-71 kubenswrapper[2578]: E1102 17:23:07.272792 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pxq56" podUID="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" Nov 02 17:23:07.646683 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.646657 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeReady" Nov 02 17:23:07.646860 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.646805 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 17:23:07.739925 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.739892 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9sr25"] Nov 02 17:23:07.741490 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.741476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.742691 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.742664 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zg8xf"] Nov 02 17:23:07.744256 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.744239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:07.760013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.759989 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kmkz5"] Nov 02 17:23:07.761412 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.761399 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.793724 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.793686 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:23:07.793908 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.793749 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:23:07.793908 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.793785 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b28rw\"" Nov 02 17:23:07.800633 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.800598 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9sr25"] Nov 02 17:23:07.803905 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.803876 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zg8xf"] Nov 02 17:23:07.831788 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.831748 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmkz5"] Nov 02 17:23:07.894015 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.893972 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:23:07.894211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894051 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:23:07.894211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894133 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:23:07.894211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894150 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:23:07.894211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894187 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:23:07.894211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894201 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:23:07.894470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894454 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2zcw\"" Nov 02 17:23:07.894526 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894471 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:23:07.894561 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.894524 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svxv2\"" Nov 02 17:23:07.898213 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6627b470-4507-4754-8d06-ef654308f832-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.898213 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019edf2b-0578-4009-8558-fdf6eb87d58d-cert\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:07.898312 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10332e6-3673-471f-bd01-99c7b5d5ac3e-config-volume\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.898312 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898235 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10332e6-3673-471f-bd01-99c7b5d5ac3e-metrics-tls\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.898312 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6627b470-4507-4754-8d06-ef654308f832-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.898411 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898388 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq65c\" (UniqueName: \"kubernetes.io/projected/d10332e6-3673-471f-bd01-99c7b5d5ac3e-kube-api-access-vq65c\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.898443 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10332e6-3673-471f-bd01-99c7b5d5ac3e-tmp-dir\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.898443 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmfb\" (UniqueName: \"kubernetes.io/projected/019edf2b-0578-4009-8558-fdf6eb87d58d-kube-api-access-rdmfb\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:07.898536 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv8lq\" (UniqueName: \"kubernetes.io/projected/6627b470-4507-4754-8d06-ef654308f832-kube-api-access-rv8lq\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.898536 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898520 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627b470-4507-4754-8d06-ef654308f832-data-volume\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.898639 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.898617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6627b470-4507-4754-8d06-ef654308f832-crio-socket\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.999403 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6627b470-4507-4754-8d06-ef654308f832-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.999403 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019edf2b-0578-4009-8558-fdf6eb87d58d-cert\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10332e6-3673-471f-bd01-99c7b5d5ac3e-config-volume\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10332e6-3673-471f-bd01-99c7b5d5ac3e-metrics-tls\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6627b470-4507-4754-8d06-ef654308f832-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq65c\" (UniqueName: \"kubernetes.io/projected/d10332e6-3673-471f-bd01-99c7b5d5ac3e-kube-api-access-vq65c\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10332e6-3673-471f-bd01-99c7b5d5ac3e-tmp-dir\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmfb\" (UniqueName: \"kubernetes.io/projected/019edf2b-0578-4009-8558-fdf6eb87d58d-kube-api-access-rdmfb\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8lq\" (UniqueName: \"kubernetes.io/projected/6627b470-4507-4754-8d06-ef654308f832-kube-api-access-rv8lq\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627b470-4507-4754-8d06-ef654308f832-data-volume\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:07.999668 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6627b470-4507-4754-8d06-ef654308f832-crio-socket\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.000108 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:07.999861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6627b470-4507-4754-8d06-ef654308f832-crio-socket\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.000285 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.000262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627b470-4507-4754-8d06-ef654308f832-data-volume\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.000344 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.000267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6627b470-4507-4754-8d06-ef654308f832-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.000344 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.000267 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10332e6-3673-471f-bd01-99c7b5d5ac3e-config-volume\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:08.000344 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.000307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10332e6-3673-471f-bd01-99c7b5d5ac3e-tmp-dir\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:08.003858 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.003832 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10332e6-3673-471f-bd01-99c7b5d5ac3e-metrics-tls\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:08.004011 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.003865 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6627b470-4507-4754-8d06-ef654308f832-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.004011 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.003933 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019edf2b-0578-4009-8558-fdf6eb87d58d-cert\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:08.056327 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.056289 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq65c\" (UniqueName: \"kubernetes.io/projected/d10332e6-3673-471f-bd01-99c7b5d5ac3e-kube-api-access-vq65c\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:08.057131 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.057114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8lq\" (UniqueName: \"kubernetes.io/projected/6627b470-4507-4754-8d06-ef654308f832-kube-api-access-rv8lq\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.057179 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.057143 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmfb\" (UniqueName: \"kubernetes.io/projected/019edf2b-0578-4009-8558-fdf6eb87d58d-kube-api-access-rdmfb\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:08.057220 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.057205 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:08.072362 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.072339 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:08.353463 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.353433 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:08.436887 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.436844 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zg8xf"] Nov 02 17:23:08.440883 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:08.440859 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod019edf2b_0578_4009_8558_fdf6eb87d58d.slice/crio-5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92 WatchSource:0}: Error finding container 5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92: Status 404 returned error can't find the container with id 5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92 Nov 02 17:23:08.487616 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.487585 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmkz5"] Nov 02 17:23:08.490490 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:08.490465 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6627b470_4507_4754_8d06_ef654308f832.slice/crio-9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1 WatchSource:0}: Error finding container 9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1: Status 404 returned error can't find the container with id 9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1 Nov 02 17:23:08.519648 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.519616 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmkz5" event={"ID":"6627b470-4507-4754-8d06-ef654308f832","Type":"ContainerStarted","Data":"9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1"} Nov 02 17:23:08.521034 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.521000 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zg8xf" event={"ID":"019edf2b-0578-4009-8558-fdf6eb87d58d","Type":"ContainerStarted","Data":"5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92"} Nov 02 17:23:08.718633 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:08.718555 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9sr25"] Nov 02 17:23:08.721557 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:08.721529 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10332e6_3673_471f_bd01_99c7b5d5ac3e.slice/crio-cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e WatchSource:0}: Error finding container cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e: Status 404 returned error can't find the container with id cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e Nov 02 17:23:09.271919 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.271883 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:09.272090 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.271894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:09.272090 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.271902 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:09.325655 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.325620 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:23:09.325655 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.325637 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:23:09.325872 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.325752 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\"" Nov 02 17:23:09.375534 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.375501 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\"" Nov 02 17:23:09.375534 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.375506 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:23:09.375534 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.375540 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:23:09.376089 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.375578 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\"" Nov 02 17:23:09.526468 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.526130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmkz5" event={"ID":"6627b470-4507-4754-8d06-ef654308f832","Type":"ContainerStarted","Data":"9a6d88dc694a60e427f1cba94db056bb92b2c50d7a1d21e71d28c68ff1d6601f"} Nov 02 17:23:09.527597 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:09.527564 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9sr25" event={"ID":"d10332e6-3673-471f-bd01-99c7b5d5ac3e","Type":"ContainerStarted","Data":"cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e"} Nov 02 17:23:10.532854 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:10.532774 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmkz5" event={"ID":"6627b470-4507-4754-8d06-ef654308f832","Type":"ContainerStarted","Data":"8bb3b21e02be52ca787db8f9405d9f3afc8ffe8c23c35c39e794b189194aa1dd"} Nov 02 17:23:11.537829 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:11.537793 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9sr25" event={"ID":"d10332e6-3673-471f-bd01-99c7b5d5ac3e","Type":"ContainerStarted","Data":"1c948f88006ed4e8dac1de68807d2e60e278fd24c868bbc7032e654fb2b7dafb"} Nov 02 17:23:11.538293 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:11.537853 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9sr25" event={"ID":"d10332e6-3673-471f-bd01-99c7b5d5ac3e","Type":"ContainerStarted","Data":"06fd0ec2306b3f78e12a1e7211a843621e460e9565894d0f7649bb9778a449f3"} Nov 02 17:23:11.538293 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:11.537919 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:12.543353 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.543320 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmkz5" event={"ID":"6627b470-4507-4754-8d06-ef654308f832","Type":"ContainerStarted","Data":"7fd99fd397d3acafeef284679853daf7043863d7af872bb3842d2113b7cd53f1"} Nov 02 17:23:12.544962 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.544933 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zg8xf" event={"ID":"019edf2b-0578-4009-8558-fdf6eb87d58d","Type":"ContainerStarted","Data":"fdf8298e587cf3e077d6c97337f3cb26f1b237bcbff009fe8123e3fdcd58421c"} Nov 02 17:23:12.661277 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.661213 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kmkz5" podStartSLOduration=2.466770486 podStartE2EDuration="5.661193807s" totalImagesPullingTime="3.194423321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:23:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:12.661144288 +0000 UTC m=+52.837345867" watchObservedRunningTime="2026-11-02 17:23:12.661193807 +0000 UTC m=+52.837395386" Nov 02 17:23:12.661949 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.661917 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-9sr25" podStartSLOduration=3.312684339 podStartE2EDuration="5.661906901s" totalImagesPullingTime="2.349222562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:11.654943936 +0000 UTC m=+51.831145513" watchObservedRunningTime="2026-11-02 17:23:12.661906901 +0000 UTC m=+52.838108511" Nov 02 17:23:12.674926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.674902 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf"] Nov 02 17:23:12.676501 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.676488 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:12.728769 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.728720 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-clgvl\"" Nov 02 17:23:12.728925 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.728908 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 17:23:12.734437 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.734410 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf"] Nov 02 17:23:12.745890 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.745864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwmxf\" (UID: \"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:12.776211 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.776166 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zg8xf" podStartSLOduration=2.40856531 podStartE2EDuration="5.776153022s" totalImagesPullingTime="3.367587712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:12.775469946 +0000 UTC m=+52.951671525" watchObservedRunningTime="2026-11-02 17:23:12.776153022 +0000 UTC m=+52.952354600" Nov 02 17:23:12.846714 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.846690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwmxf\" (UID: \"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:12.849234 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.849209 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwmxf\" (UID: \"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:12.985459 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:12.985422 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:13.348885 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:13.348857 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf"] Nov 02 17:23:13.352483 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:13.352453 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31ce7db_987c_4f05_9a4b_fbfd2fa0b113.slice/crio-53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd WatchSource:0}: Error finding container 53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd: Status 404 returned error can't find the container with id 53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd Nov 02 17:23:13.548496 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:13.548460 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" event={"ID":"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113","Type":"ContainerStarted","Data":"53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd"} Nov 02 17:23:15.558549 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:15.558515 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" event={"ID":"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113","Type":"ContainerStarted","Data":"48d7969f4ac3b6507a39e0d2919de2faea4f241f78f60ec8567eb7c6b4d5e72d"} Nov 02 17:23:15.558998 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:15.558760 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:15.563290 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:15.563272 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:15.670777 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:15.670700 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" podStartSLOduration=2.176089314 podStartE2EDuration="3.670688185s" totalImagesPullingTime="1.494598871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:15.670136248 +0000 UTC m=+55.846337841" watchObservedRunningTime="2026-11-02 17:23:15.670688185 +0000 UTC m=+55.846889762" Nov 02 17:23:16.482231 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:16.482205 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:17.322179 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.322139 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8"] Nov 02 17:23:17.325191 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.325168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.377229 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.377199 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:23:17.377507 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.377479 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:23:17.377792 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.377771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.377853 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.377829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fsz8\" (UniqueName: \"kubernetes.io/projected/34b5f718-d276-44f2-b1b5-f9bf961147ed-kube-api-access-4fsz8\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.377922 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.377904 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.377968 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.377948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b5f718-d276-44f2-b1b5-f9bf961147ed-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.426685 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.426653 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b2f84\"" Nov 02 17:23:17.426685 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.426674 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:23:17.426906 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.426802 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:23:17.426906 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.426883 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:23:17.427780 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.427764 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8"] Nov 02 17:23:17.478386 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.478347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsz8\" (UniqueName: \"kubernetes.io/projected/34b5f718-d276-44f2-b1b5-f9bf961147ed-kube-api-access-4fsz8\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.478553 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.478395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.478553 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.478441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b5f718-d276-44f2-b1b5-f9bf961147ed-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.478553 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.478508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.479228 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.479142 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b5f718-d276-44f2-b1b5-f9bf961147ed-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.481132 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.481108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.481231 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.481104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.535210 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.535175 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fsz8\" (UniqueName: \"kubernetes.io/projected/34b5f718-d276-44f2-b1b5-f9bf961147ed-kube-api-access-4fsz8\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:17.638214 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:17.638177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:18.003591 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:18.003462 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8"] Nov 02 17:23:18.006912 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:18.006887 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b5f718_d276_44f2_b1b5_f9bf961147ed.slice/crio-5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1 WatchSource:0}: Error finding container 5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1: Status 404 returned error can't find the container with id 5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1 Nov 02 17:23:18.570091 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:18.570055 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" event={"ID":"34b5f718-d276-44f2-b1b5-f9bf961147ed","Type":"ContainerStarted","Data":"5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1"} Nov 02 17:23:20.577926 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:20.577885 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" event={"ID":"34b5f718-d276-44f2-b1b5-f9bf961147ed","Type":"ContainerStarted","Data":"86d7bf1a6f2ed9f60dd19ea32bcaa8ae00ae91845fbeaa9a771cae02497a4d00"} Nov 02 17:23:20.578424 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:20.577940 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" event={"ID":"34b5f718-d276-44f2-b1b5-f9bf961147ed","Type":"ContainerStarted","Data":"a1152edd2cef2645c6e33c1b0aa7db256244150107637002f9fa96c9d30942c5"} Nov 02 17:23:20.699506 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:20.699438 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" podStartSLOduration=1.9670300520000001 podStartE2EDuration="3.699421883s" totalImagesPullingTime="1.732391831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:20.69868105 +0000 UTC m=+60.874882627" watchObservedRunningTime="2026-11-02 17:23:20.699421883 +0000 UTC m=+60.875623500" Nov 02 17:23:21.547181 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:21.547150 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:23.779721 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.779689 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97"] Nov 02 17:23:23.783828 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.783813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.821895 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.821861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15153aa-78ce-426a-9b63-5a4489d8fc45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.822024 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.821908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhzj\" (UniqueName: \"kubernetes.io/projected/d15153aa-78ce-426a-9b63-5a4489d8fc45-kube-api-access-fxhzj\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.822024 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.821928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.822104 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.822055 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.885593 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.885559 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:23:23.885593 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.885579 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8lnng\"" Nov 02 17:23:23.885593 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.885572 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 17:23:23.886637 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.886622 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq"] Nov 02 17:23:23.889590 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.889573 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97"] Nov 02 17:23:23.889686 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.889676 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.923176 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhzj\" (UniqueName: \"kubernetes.io/projected/d15153aa-78ce-426a-9b63-5a4489d8fc45-kube-api-access-fxhzj\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923408 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923441 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361abe17-e8b7-4695-ae30-8cd0ccc06017-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.923470 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15153aa-78ce-426a-9b63-5a4489d8fc45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.923952 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.923490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m829p\" (UniqueName: \"kubernetes.io/projected/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-api-access-m829p\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:23.924508 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.924487 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15153aa-78ce-426a-9b63-5a4489d8fc45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.926441 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.926417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.926559 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.926537 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:23.937366 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.937339 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jmf49"] Nov 02 17:23:23.940489 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.940474 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:23.943390 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.943366 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:23:23.943809 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.943793 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9fqjw\"" Nov 02 17:23:23.943860 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.943803 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:23:23.952479 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.952455 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq"] Nov 02 17:23:23.981275 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:23.981247 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhzj\" (UniqueName: \"kubernetes.io/projected/d15153aa-78ce-426a-9b63-5a4489d8fc45-kube-api-access-fxhzj\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:24.024356 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024539 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrj4d\" (UniqueName: \"kubernetes.io/projected/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-kube-api-access-mrj4d\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024539 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.024539 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024507 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-root\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024539 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024526 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-wtmp\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024660 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-sys\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024660 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361abe17-e8b7-4695-ae30-8cd0ccc06017-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.024825 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m829p\" (UniqueName: \"kubernetes.io/projected/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-api-access-m829p\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.024871 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.024904 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-textfile\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024938 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-tls\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.024982 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.024957 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.025094 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.025061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.025094 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.025087 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361abe17-e8b7-4695-ae30-8cd0ccc06017-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.025283 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.025110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-metrics-client-ca\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.025283 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.025147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.025378 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.025362 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.025550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.025533 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.027461 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.027440 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.044461 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.044382 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:23:24.045315 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.045297 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:23:24.045372 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.045334 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:23:24.045410 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.045394 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv5xv\"" Nov 02 17:23:24.045489 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.045478 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:23:24.062550 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.062524 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.084452 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.084425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m829p\" (UniqueName: \"kubernetes.io/projected/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-api-access-m829p\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.098289 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.098254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:24.126319 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-metrics-client-ca\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126319 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126504 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126504 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj4d\" (UniqueName: \"kubernetes.io/projected/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-kube-api-access-mrj4d\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126504 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-root\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126504 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-wtmp\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126504 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126491 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-sys\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126793 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-textfile\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126793 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126559 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-root\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126793 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126585 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-tls\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.126793 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126692 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-sys\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.127008 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126839 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-wtmp\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.127008 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.126988 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-metrics-client-ca\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.127109 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.127028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.127109 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.127046 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-textfile\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.128715 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.128693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.128869 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.128851 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-tls\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.184049 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.184019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj4d\" (UniqueName: \"kubernetes.io/projected/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-kube-api-access-mrj4d\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.204384 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.204354 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:24.259306 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.259272 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:24.271540 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:24.271511 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a23d348_6b2d_4ded_9ff9_543a1e01f6cf.slice/crio-58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9 WatchSource:0}: Error finding container 58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9: Status 404 returned error can't find the container with id 58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9 Nov 02 17:23:24.471890 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.471817 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97"] Nov 02 17:23:24.474116 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:24.474090 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15153aa_78ce_426a_9b63_5a4489d8fc45.slice/crio-42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10 WatchSource:0}: Error finding container 42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10: Status 404 returned error can't find the container with id 42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10 Nov 02 17:23:24.578511 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.578487 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq"] Nov 02 17:23:24.581638 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:24.581613 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361abe17_e8b7_4695_ae30_8cd0ccc06017.slice/crio-bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2 WatchSource:0}: Error finding container bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2: Status 404 returned error can't find the container with id bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2 Nov 02 17:23:24.593460 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.593435 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmf49" event={"ID":"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf","Type":"ContainerStarted","Data":"58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9"} Nov 02 17:23:24.594638 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.594615 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" event={"ID":"361abe17-e8b7-4695-ae30-8cd0ccc06017","Type":"ContainerStarted","Data":"bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2"} Nov 02 17:23:24.596445 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.596427 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" event={"ID":"d15153aa-78ce-426a-9b63-5a4489d8fc45","Type":"ContainerStarted","Data":"4579799391ca35d2154939700d06e4ff659678e84b8f67fd3c1c76860f141a16"} Nov 02 17:23:24.596526 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.596464 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" event={"ID":"d15153aa-78ce-426a-9b63-5a4489d8fc45","Type":"ContainerStarted","Data":"d0d80cc4bf01a6e95fdd84f838720c53b2b296bfd9449a599f17158e2024860e"} Nov 02 17:23:24.596526 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:24.596486 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" event={"ID":"d15153aa-78ce-426a-9b63-5a4489d8fc45","Type":"ContainerStarted","Data":"42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10"} Nov 02 17:23:25.235521 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.235483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:25.236000 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.235564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:25.286801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.286727 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:23:25.286801 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.286758 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:23:25.298345 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.298317 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:25.298458 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.298356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:25.336839 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.336795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:25.388293 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.388261 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:23:25.448350 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.448313 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:23:25.460536 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.460502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:25.535292 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.535206 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\"" Nov 02 17:23:25.539899 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.539870 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\"" Nov 02 17:23:25.544614 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.544332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:25.544856 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.544822 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\"" Nov 02 17:23:25.549649 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.549625 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:25.554899 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.554875 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:25.936542 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.936461 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hh9rw"] Nov 02 17:23:25.985920 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:25.985857 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd31aff_608d_4200_b76d_61f4c43e8a99.slice/crio-f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa WatchSource:0}: Error finding container f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa: Status 404 returned error can't find the container with id f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa Nov 02 17:23:25.987419 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.987400 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pxq56"] Nov 02 17:23:25.988663 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:25.988641 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tlgwq"] Nov 02 17:23:25.996869 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:25.996838 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf5ced3_8a35_47bb_a042_f7bfb1b5d60e.slice/crio-af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d WatchSource:0}: Error finding container af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d: Status 404 returned error can't find the container with id af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d Nov 02 17:23:26.183350 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.183319 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:23:26.188993 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.188970 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.244465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244333 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:23:26.244465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244402 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:23:26.244465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244423 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:23:26.244465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244441 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hmltn\"" Nov 02 17:23:26.244465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244457 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:23:26.244465 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244442 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:23:26.245197 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.244491 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:23:26.245961 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.245927 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246074 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.245965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246074 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246074 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246051 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246187 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246314 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.246857 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.246327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.343577 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.343541 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:23:26.344288 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.344260 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:23:26.344426 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.344366 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:23:26.345009 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.344985 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:23:26.346760 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.346862 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.346862 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346829 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.346862 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346855 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.346977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347013 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.347006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347241 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.347031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347241 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.347054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347241 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.347101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347241 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.347134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.347241 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.347158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.348160 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.348132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.348255 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.348184 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.350751 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.350678 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.351436 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.351411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.351538 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.351477 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.351538 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.351517 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.351784 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.351754 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.352830 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.352788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.353050 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.352873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.353170 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.353084 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.353615 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.353598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.403684 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.403618 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.527765 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.527717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:26.607940 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.607906 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxq56" event={"ID":"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e","Type":"ContainerStarted","Data":"af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d"} Nov 02 17:23:26.610502 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.610469 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" event={"ID":"361abe17-e8b7-4695-ae30-8cd0ccc06017","Type":"ContainerStarted","Data":"5b9d4df4a88d3f6c9d0c0fd46dc8d1e218b265e634950eca5c8f92dba34debda"} Nov 02 17:23:26.610567 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.610540 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" event={"ID":"361abe17-e8b7-4695-ae30-8cd0ccc06017","Type":"ContainerStarted","Data":"32830662c4fcfca78f36c101bc668dca52a7b27c0c5253c4996235191c03a17f"} Nov 02 17:23:26.611667 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.611648 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tlgwq" event={"ID":"9021558c-9c68-4634-a371-3172c24a2d85","Type":"ContainerStarted","Data":"cb213337ded4fd98e6ee65e89cdb24c47d0accbd42f5bf13b8288c73eff9ffe3"} Nov 02 17:23:26.613680 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.613653 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" event={"ID":"d15153aa-78ce-426a-9b63-5a4489d8fc45","Type":"ContainerStarted","Data":"6d610d3eabd18da1cab9770a647b4310d33479e3c87d2943afbf5d1ead97afa4"} Nov 02 17:23:26.614781 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.614758 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hh9rw" event={"ID":"bcd31aff-608d-4200-b76d-61f4c43e8a99","Type":"ContainerStarted","Data":"f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa"} Nov 02 17:23:26.616515 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.616491 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" containerID="43acd66ff34fbb30c51c6a47fe2374eb2ed63671ec28d2625f00d3cdebe2ca82" exitCode=0 Nov 02 17:23:26.616603 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.616523 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmf49" event={"ID":"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf","Type":"ContainerDied","Data":"43acd66ff34fbb30c51c6a47fe2374eb2ed63671ec28d2625f00d3cdebe2ca82"} Nov 02 17:23:26.730638 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.730593 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" podStartSLOduration=2.184311201 podStartE2EDuration="3.730580399s" totalImagesPullingTime="1.546269198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:26.729970762 +0000 UTC m=+66.906172333" watchObservedRunningTime="2026-11-02 17:23:26.730580399 +0000 UTC m=+66.906781977" Nov 02 17:23:26.963027 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:26.962955 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:23:27.625687 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:27.625057 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmf49" event={"ID":"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf","Type":"ContainerStarted","Data":"edb1aaf50be1f204bda9cc579eec2b3caceb179f37b1fbf0f580b56c9d4e1f76"} Nov 02 17:23:27.625687 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:27.625130 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jmf49" event={"ID":"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf","Type":"ContainerStarted","Data":"ec6e44f8bda157609818ae149644cc12a04ccfa556a66d2c5f701e3eb0e56473"} Nov 02 17:23:27.627962 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:27.627932 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" event={"ID":"361abe17-e8b7-4695-ae30-8cd0ccc06017","Type":"ContainerStarted","Data":"8639e3f5f03bc8c253bff9a6cdffe01c1d25d00e2ba5202b5381e718941d0ed8"} Nov 02 17:23:27.629622 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:27.629539 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"94acfe4437d84c18f686cc30b375026f945a6a93da63a79b6ebb88def859a6bb"} Nov 02 17:23:27.743675 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:27.743630 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jmf49" podStartSLOduration=3.112608529 podStartE2EDuration="4.743617259s" totalImagesPullingTime="1.532247068s" totalInitContainerRuntime="98.761662ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:27.741776691 +0000 UTC m=+67.917978270" watchObservedRunningTime="2026-11-02 17:23:27.743617259 +0000 UTC m=+67.919818837" Nov 02 17:23:28.607207 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.607144 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" podStartSLOduration=3.7607581359999998 podStartE2EDuration="5.607120538s" totalImagesPullingTime="1.846362402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:27.858432073 +0000 UTC m=+68.034633650" watchObservedRunningTime="2026-11-02 17:23:28.607120538 +0000 UTC m=+68.783322118" Nov 02 17:23:28.607433 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.607416 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gjj99"] Nov 02 17:23:28.611350 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.610815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.636700 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.636661 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hh9rw" event={"ID":"bcd31aff-608d-4200-b76d-61f4c43e8a99","Type":"ContainerStarted","Data":"0ff66f0ea93e100f73aff95d6ef3fa346473fbee5ee90ca37d0261caf11ca89b"} Nov 02 17:23:28.637121 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.636727 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hh9rw" event={"ID":"bcd31aff-608d-4200-b76d-61f4c43e8a99","Type":"ContainerStarted","Data":"3481b36f22382402dc6346a8756c77bce5b0ade962510e2672c4ea5917f60709"} Nov 02 17:23:28.669838 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.669696 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jbq9x\"" Nov 02 17:23:28.669838 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.669810 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:23:28.670784 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.670755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.670918 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.670796 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gjj99"] Nov 02 17:23:28.670972 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.670951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvb2\" (UniqueName: \"kubernetes.io/projected/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-kube-api-access-8rvb2\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.772208 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.772159 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.772388 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.772309 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvb2\" (UniqueName: \"kubernetes.io/projected/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-kube-api-access-8rvb2\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.775558 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.775520 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.818108 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.818076 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f564b5ffb-5scd2"] Nov 02 17:23:28.822561 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.821670 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.830991 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.830962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvb2\" (UniqueName: \"kubernetes.io/projected/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-kube-api-access-8rvb2\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.840099 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.840053 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hh9rw" podStartSLOduration=66.119027686 podStartE2EDuration="1m7.840040437s" totalImagesPullingTime="1.721012751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:28.838925675 +0000 UTC m=+69.015127250" watchObservedRunningTime="2026-11-02 17:23:28.840040437 +0000 UTC m=+69.016242012" Nov 02 17:23:28.873214 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-metrics-server-audit-profiles\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.873384 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.873384 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809d0903-74ab-4c73-a6db-5811776a58ec-audit-log\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.873490 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-tls\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.873490 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-client-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.873584 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873506 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-client-certs\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.873584 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.873533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vk5g\" (UniqueName: \"kubernetes.io/projected/809d0903-74ab-4c73-a6db-5811776a58ec-kube-api-access-5vk5g\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.874692 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.874664 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zwtpn\"" Nov 02 17:23:28.874692 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.874682 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ar746rs3icng6\"" Nov 02 17:23:28.874865 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.874672 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 17:23:28.874865 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.874719 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 17:23:28.877243 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.877220 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:23:28.878723 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.878695 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f564b5ffb-5scd2"] Nov 02 17:23:28.925389 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.925130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974223 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974580 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809d0903-74ab-4c73-a6db-5811776a58ec-audit-log\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974631 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-tls\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-client-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-client-certs\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vk5g\" (UniqueName: \"kubernetes.io/projected/809d0903-74ab-4c73-a6db-5811776a58ec-kube-api-access-5vk5g\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.974768 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.974712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-metrics-server-audit-profiles\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.976521 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.975455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-metrics-server-audit-profiles\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.976521 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.975861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.976521 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.976039 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809d0903-74ab-4c73-a6db-5811776a58ec-audit-log\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.979330 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.979307 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-tls\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.987254 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.987229 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-client-certs\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:28.990296 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:28.986725 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-client-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:29.035001 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.034970 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vk5g\" (UniqueName: \"kubernetes.io/projected/809d0903-74ab-4c73-a6db-5811776a58ec-kube-api-access-5vk5g\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:29.147798 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.147688 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:29.312189 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.312155 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gjj99"] Nov 02 17:23:29.316307 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:29.316274 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda924f2fb_68fd_4d19_bb5d_26eb25dcd9bd.slice/crio-dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc WatchSource:0}: Error finding container dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc: Status 404 returned error can't find the container with id dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc Nov 02 17:23:29.530779 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.530671 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f564b5ffb-5scd2"] Nov 02 17:23:29.536846 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:29.536813 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809d0903_74ab_4c73_a6db_5811776a58ec.slice/crio-51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0 WatchSource:0}: Error finding container 51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0: Status 404 returned error can't find the container with id 51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0 Nov 02 17:23:29.644614 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.644571 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" event={"ID":"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd","Type":"ContainerStarted","Data":"dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc"} Nov 02 17:23:29.646782 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.646755 2578 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3" exitCode=0 Nov 02 17:23:29.647050 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.647030 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3"} Nov 02 17:23:29.648784 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.648756 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" event={"ID":"809d0903-74ab-4c73-a6db-5811776a58ec","Type":"ContainerStarted","Data":"51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0"} Nov 02 17:23:29.993901 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.992577 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-wwch5"] Nov 02 17:23:29.995606 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:29.995581 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.053302 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.053217 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-z497l\"" Nov 02 17:23:30.055568 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.055538 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wwch5"] Nov 02 17:23:30.059448 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.059421 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:23:30.059574 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.059462 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:23:30.085916 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.085879 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b463fd2-4b01-4441-8d41-dbbe0a440a22-tmp\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.086078 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.085931 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws42m\" (UniqueName: \"kubernetes.io/projected/4b463fd2-4b01-4441-8d41-dbbe0a440a22-kube-api-access-ws42m\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.187441 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.187409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b463fd2-4b01-4441-8d41-dbbe0a440a22-tmp\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.187613 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.187463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws42m\" (UniqueName: \"kubernetes.io/projected/4b463fd2-4b01-4441-8d41-dbbe0a440a22-kube-api-access-ws42m\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.187865 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.187841 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b463fd2-4b01-4441-8d41-dbbe0a440a22-tmp\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.249646 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.249176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws42m\" (UniqueName: \"kubernetes.io/projected/4b463fd2-4b01-4441-8d41-dbbe0a440a22-kube-api-access-ws42m\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.310502 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.310431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:30.656598 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.656252 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tlgwq" event={"ID":"9021558c-9c68-4634-a371-3172c24a2d85","Type":"ContainerStarted","Data":"c09d4d6080c15773b4ee54e2fab3a0c312a816569172705e661e4dfb88d0255d"} Nov 02 17:23:30.656598 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.656379 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:30.746519 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.746481 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-wwch5"] Nov 02 17:23:30.753764 ip-10-0-128-71 kubenswrapper[2578]: W1102 17:23:30.753698 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b463fd2_4b01_4441_8d41_dbbe0a440a22.slice/crio-d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52 WatchSource:0}: Error finding container d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52: Status 404 returned error can't find the container with id d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52 Nov 02 17:23:30.770095 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:30.770047 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tlgwq" podStartSLOduration=66.556735173 podStartE2EDuration="1m10.770032724s" totalImagesPullingTime="4.213297551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:30.768975987 +0000 UTC m=+70.945177568" watchObservedRunningTime="2026-11-02 17:23:30.770032724 +0000 UTC m=+70.946234281" Nov 02 17:23:31.662232 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:31.662202 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wwch5" event={"ID":"4b463fd2-4b01-4441-8d41-dbbe0a440a22","Type":"ContainerStarted","Data":"d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52"} Nov 02 17:23:31.665107 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:31.665079 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" event={"ID":"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd","Type":"ContainerStarted","Data":"0cb462544e9b243f71ee0a4587c82adce90d757f0c9b66f39d241ead531636cb"} Nov 02 17:23:31.665364 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:31.665345 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:31.682885 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:31.671135 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:31.790888 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:31.790363 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" podStartSLOduration=1.765670625 podStartE2EDuration="3.790343004s" totalImagesPullingTime="2.024672379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:31.789237315 +0000 UTC m=+71.965438895" watchObservedRunningTime="2026-11-02 17:23:31.790343004 +0000 UTC m=+71.966544587" Nov 02 17:23:32.674838 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.671177 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pxq56" event={"ID":"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e","Type":"ContainerStarted","Data":"bef6255ede0b54099d819ce30f048d678ea312d872fbef1dabc304f8d662f12b"} Nov 02 17:23:32.675431 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.675309 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee"} Nov 02 17:23:32.675431 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.675386 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639"} Nov 02 17:23:32.675542 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.675431 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4"} Nov 02 17:23:32.675542 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.675473 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888"} Nov 02 17:23:32.675542 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.675514 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d"} Nov 02 17:23:32.678327 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.678294 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" event={"ID":"809d0903-74ab-4c73-a6db-5811776a58ec","Type":"ContainerStarted","Data":"46d829c28bde93606bb0573d185a5a3ce98c34d7005f26cc15ca4b0d9213a091"} Nov 02 17:23:32.787651 ip-10-0-128-71 kubenswrapper[2578]: I1102 17:23:32.785509 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pxq56" podStartSLOduration=65.26291176 podStartE2EDuration="1m11.785489281s" totalImagesPullingTime="6.522577521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:22:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:32.784075281 +0000 UTC m=+72.960276862" watchObservedRunningTime="2026-11-02 17:23:32.785489281 +0000 UTC m=+72.961690860" Nov 02 17:23:32.859950 ip-10-0-128-71 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 17:23:32.876226 ip-10-0-128-71 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 17:23:32.876454 ip-10-0-128-71 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 17:23:32.876513 ip-10-0-128-71 systemd[1]: kubelet.service: Consumed 4.278s CPU time. Nov 02 17:23:32.892315 ip-10-0-128-71 systemd[1]: Starting Kubernetes Kubelet... Nov 02 17:23:32.960384 ip-10-0-128-71 kubenswrapper[5647]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:23:32.960384 ip-10-0-128-71 kubenswrapper[5647]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 17:23:32.960384 ip-10-0-128-71 kubenswrapper[5647]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:23:32.960384 ip-10-0-128-71 kubenswrapper[5647]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962725 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962857 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962862 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962866 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962871 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962875 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962881 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962885 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962889 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962893 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962897 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962901 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962905 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962909 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962913 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962917 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962922 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962926 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962930 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.964766 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962934 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962939 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962943 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962948 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962952 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962956 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962960 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962965 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962970 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962974 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962978 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962983 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962987 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962991 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962995 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.962999 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963003 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963011 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963017 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.965691 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963022 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963026 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963030 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963035 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963038 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963043 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963047 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963051 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963055 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963059 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963063 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963067 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963071 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963075 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963079 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963084 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963088 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963094 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963100 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.966525 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963105 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963121 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963125 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963129 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963133 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963137 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963141 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963145 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963149 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963153 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963157 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963161 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963165 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963169 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963173 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963177 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963181 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963186 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963190 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963194 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.967372 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963198 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963202 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963206 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963210 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963214 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963218 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963223 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963228 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963232 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963236 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963240 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963244 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963248 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963252 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963258 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963262 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963266 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963271 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963275 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.968204 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963279 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963283 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963287 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963291 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963295 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963301 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963305 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963309 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963312 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.963317 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964492 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964505 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964510 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964515 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964519 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964523 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964527 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964531 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964539 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964544 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.969042 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964549 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964553 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964557 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964561 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964565 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964568 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964572 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964576 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964581 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964585 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964589 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964593 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964597 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964601 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964605 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964608 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964613 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964616 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964620 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.971999 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964624 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964628 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964632 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964636 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964640 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964644 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964648 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964652 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964656 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964660 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964665 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964670 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964674 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964678 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964682 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964686 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964690 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964694 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964698 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.972810 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964702 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964708 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964713 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964725 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964749 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964754 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964758 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964763 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964767 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964771 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964774 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964778 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964782 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964787 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964797 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964804 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964808 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964812 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964819 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.973506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964825 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964829 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964834 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964838 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964844 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964849 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964853 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964858 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964862 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964868 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964872 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964876 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964880 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964885 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964890 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964894 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964898 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964902 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964905 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.973993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964908 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964911 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964913 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964916 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964919 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964921 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964924 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964926 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964930 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964932 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964935 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964937 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964940 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964942 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964945 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964948 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964951 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964953 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964956 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.964959 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.974488 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965774 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965785 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965790 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965798 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965804 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965808 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965813 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965817 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965821 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965825 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965831 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965835 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965839 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965843 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965848 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965852 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965856 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965860 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965864 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.974990 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965867 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965870 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965873 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965876 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965879 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965882 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965885 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965887 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965890 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965892 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965895 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965897 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965900 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965902 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965905 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965908 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965910 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965913 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965915 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965917 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.975443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965920 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965922 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965925 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965927 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965930 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965933 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965936 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965939 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965941 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965944 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965947 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965949 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965952 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965958 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965960 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965963 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965965 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965968 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965970 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965973 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.976059 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965975 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965978 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965981 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965984 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965986 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965989 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965991 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965994 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965996 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.965999 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966002 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966005 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966007 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966009 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966012 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966014 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966016 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966020 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966022 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966025 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.976908 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966028 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966030 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966033 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966035 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966038 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966040 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966043 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966045 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966047 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966050 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966052 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966055 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966057 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966060 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966062 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966067 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966070 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966073 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966075 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.977678 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966078 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966080 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966083 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966086 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966089 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966094 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966097 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966100 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966222 5647 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966233 5647 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966241 5647 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966247 5647 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966251 5647 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966254 5647 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966259 5647 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966263 5647 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966266 5647 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966269 5647 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966273 5647 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966276 5647 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966281 5647 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966285 5647 flags.go:64] FLAG: --cgroup-root="" Nov 02 17:23:32.978411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966288 5647 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966291 5647 flags.go:64] FLAG: --client-ca-file="" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966294 5647 flags.go:64] FLAG: --cloud-provider="external" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966297 5647 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966300 5647 flags.go:64] FLAG: --cluster-domain="" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966303 5647 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966310 5647 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966313 5647 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966316 5647 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966320 5647 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966323 5647 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966326 5647 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966330 5647 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966333 5647 flags.go:64] FLAG: --contention-profiling="false" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966336 5647 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966339 5647 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966342 5647 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966345 5647 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966350 5647 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966353 5647 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966356 5647 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966358 5647 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966362 5647 flags.go:64] FLAG: --enable-server="true" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966365 5647 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966369 5647 flags.go:64] FLAG: --event-burst="100" Nov 02 17:23:32.979115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966373 5647 flags.go:64] FLAG: --event-qps="50" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966375 5647 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966378 5647 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966381 5647 flags.go:64] FLAG: --eviction-hard="" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966385 5647 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966389 5647 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966393 5647 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966395 5647 flags.go:64] FLAG: --eviction-soft="" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966399 5647 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966401 5647 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966404 5647 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966407 5647 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966410 5647 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966414 5647 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966417 5647 flags.go:64] FLAG: --feature-gates="" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966421 5647 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966424 5647 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966428 5647 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966431 5647 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966434 5647 flags.go:64] FLAG: --healthz-port="10248" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966437 5647 flags.go:64] FLAG: --help="false" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966440 5647 flags.go:64] FLAG: --hostname-override="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966443 5647 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966446 5647 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 17:23:32.979877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966449 5647 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966452 5647 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966455 5647 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966458 5647 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966461 5647 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966463 5647 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966467 5647 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966470 5647 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966473 5647 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966476 5647 flags.go:64] FLAG: --kube-reserved="" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966479 5647 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966481 5647 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966484 5647 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966489 5647 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966492 5647 flags.go:64] FLAG: --lock-file="" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966495 5647 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966498 5647 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966501 5647 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966506 5647 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966509 5647 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966512 5647 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966517 5647 flags.go:64] FLAG: --logging-format="text" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966520 5647 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966523 5647 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 17:23:32.980777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966526 5647 flags.go:64] FLAG: --manifest-url="" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966529 5647 flags.go:64] FLAG: --manifest-url-header="" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966534 5647 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966537 5647 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966541 5647 flags.go:64] FLAG: --max-pods="110" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966544 5647 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966547 5647 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966550 5647 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966553 5647 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966556 5647 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966559 5647 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966561 5647 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966569 5647 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966572 5647 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966575 5647 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966579 5647 flags.go:64] FLAG: --pod-cidr="" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966581 5647 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966584 5647 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966587 5647 flags.go:64] FLAG: --pods-per-core="0" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966590 5647 flags.go:64] FLAG: --port="10250" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966593 5647 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966597 5647 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0e5ee1eb0fcde3293" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966600 5647 flags.go:64] FLAG: --qos-reserved="" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966603 5647 flags.go:64] FLAG: --read-only-port="10255" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966606 5647 flags.go:64] FLAG: --register-node="true" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966609 5647 flags.go:64] FLAG: --register-with-taints="" Nov 02 17:23:32.981478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966616 5647 flags.go:64] FLAG: --registry-burst="10" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966619 5647 flags.go:64] FLAG: --registry-qps="5" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966621 5647 flags.go:64] FLAG: --reserved-cpus="" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966625 5647 flags.go:64] FLAG: --reserved-memory="" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966629 5647 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966632 5647 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966635 5647 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966638 5647 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966641 5647 flags.go:64] FLAG: --runonce="false" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966644 5647 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966648 5647 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966650 5647 flags.go:64] FLAG: --seccomp-default="false" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966653 5647 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966656 5647 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966659 5647 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966662 5647 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966665 5647 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966668 5647 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966671 5647 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966674 5647 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966677 5647 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966680 5647 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966683 5647 flags.go:64] FLAG: --system-cgroups="" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966686 5647 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966691 5647 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 17:23:32.982233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966694 5647 flags.go:64] FLAG: --tls-cert-file="" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966697 5647 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966701 5647 flags.go:64] FLAG: --tls-min-version="" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966704 5647 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966707 5647 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966710 5647 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966712 5647 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966715 5647 flags.go:64] FLAG: --v="2" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966720 5647 flags.go:64] FLAG: --version="false" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966724 5647 flags.go:64] FLAG: --vmodule="" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966745 5647 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.966748 5647 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966887 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966891 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966895 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966898 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966900 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966903 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966906 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966908 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966911 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966913 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.982849 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966917 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966921 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966925 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966928 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966930 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966933 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966935 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966938 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966941 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966944 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966946 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966949 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966953 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966956 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966958 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966961 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966963 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966966 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966969 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.983385 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966973 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966978 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966981 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966984 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966987 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966990 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966993 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966995 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.966997 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967000 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967002 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967005 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967007 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967010 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967012 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967015 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967017 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967020 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967023 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967025 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.983986 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967028 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967030 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967033 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967036 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967039 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967044 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967047 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967049 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967052 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967055 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967057 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967060 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967062 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967066 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967068 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967071 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967073 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967076 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967078 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967081 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.984544 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967099 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967104 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967107 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967110 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967112 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967115 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967117 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967120 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967122 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967125 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967127 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967130 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967133 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967135 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967138 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967141 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967144 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967147 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967150 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.985443 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967153 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967155 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967158 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967160 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967163 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967165 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967169 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967171 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967173 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967176 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967179 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967181 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967184 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967186 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967189 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967191 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967193 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.986160 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967196 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.967222 5647 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967260 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967264 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967267 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967270 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967273 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967275 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967278 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967281 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967283 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967286 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967289 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967291 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967295 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967298 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.986784 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967300 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967303 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967305 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967307 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967310 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967313 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967316 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967318 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967321 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967324 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967326 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967328 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967331 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967333 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967336 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967338 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967341 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967343 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967346 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967348 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.987393 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967351 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967353 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967355 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967358 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967360 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967363 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967365 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967368 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967370 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967373 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967377 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967379 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967382 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967385 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967387 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967390 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967392 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967396 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967399 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967401 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.988118 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967403 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967405 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967408 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967410 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967413 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967415 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967417 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967420 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967422 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967425 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967427 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967430 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967432 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967436 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967439 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967442 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967444 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967447 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967449 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.988622 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967451 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967454 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967456 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967461 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967463 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967466 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967468 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967471 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967473 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967475 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967479 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967481 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967485 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967488 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967491 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967493 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967496 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967498 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967501 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.989191 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967503 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967506 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967508 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967511 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967513 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967515 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967518 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967521 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967523 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967526 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967528 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967530 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967533 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.967535 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.974330 5647 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.974349 5647 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.974367 5647 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.974372 5647 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974433 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.989944 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974438 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974441 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974444 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974447 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974450 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974453 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974456 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974458 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974461 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974463 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974466 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974469 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974471 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974475 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974478 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974480 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974483 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974485 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974488 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.990550 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974491 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974494 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974497 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974499 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974502 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974505 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974508 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974511 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974513 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974516 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974519 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974521 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974524 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974527 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974529 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974532 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974534 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974537 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974540 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974542 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.991029 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974545 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974547 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974549 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974552 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974554 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974557 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974560 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974563 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974565 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974568 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974570 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974574 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974577 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974579 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974582 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974586 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974590 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974594 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974597 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974600 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.991541 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974602 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974605 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974607 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974610 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974612 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974615 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974618 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974620 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974623 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974625 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974628 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974630 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974633 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974636 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974638 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974641 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974644 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974646 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974648 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974668 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.994774 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974671 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974674 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974677 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974680 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974682 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974685 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974688 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974690 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974693 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974695 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974698 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974700 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974703 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974705 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974707 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974711 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974715 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974718 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974721 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.995333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974723 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974726 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974746 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974750 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974752 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974754 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974757 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.974786 5647 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974939 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974944 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974948 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974951 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974954 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974958 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974961 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974964 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.995901 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974966 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974969 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974971 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974974 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974977 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974979 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974981 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974984 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974987 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974989 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974991 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974994 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974997 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.974999 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975002 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975005 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975007 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975010 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975013 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975015 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:32.996340 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975019 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975021 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975024 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975026 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975029 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975033 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975037 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975040 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975043 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975046 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975049 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975051 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975053 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975056 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975059 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975061 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975076 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975080 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975083 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:32.996950 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975087 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975090 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975092 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975095 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975097 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975100 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975102 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975105 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975108 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975110 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975116 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975119 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975122 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975124 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975127 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975130 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975132 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975135 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975137 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975140 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:32.997481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975142 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975145 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975147 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975150 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975153 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975155 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975158 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975161 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975163 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975166 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975168 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975170 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975173 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975175 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975178 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975180 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975183 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975185 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975189 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:32.998110 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975192 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975195 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975198 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975202 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975205 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975209 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975211 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975215 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975218 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975221 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975223 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975226 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975228 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975231 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975233 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975236 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975238 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975240 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975244 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.998697 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975247 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.975269 5647 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975377 5647 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975385 5647 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975390 5647 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975394 5647 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975397 5647 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975402 5647 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975406 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975410 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975414 5647 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975418 5647 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975422 5647 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975427 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975431 5647 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975434 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 17:23:32.999491 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975441 5647 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975445 5647 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975450 5647 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975454 5647 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975458 5647 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975462 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975466 5647 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975470 5647 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975474 5647 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975478 5647 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975482 5647 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975485 5647 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975487 5647 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975490 5647 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975493 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975495 5647 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975500 5647 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975503 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975506 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975508 5647 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 17:23:33.000122 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975511 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975513 5647 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975516 5647 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975518 5647 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975522 5647 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975526 5647 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975528 5647 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975531 5647 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975534 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975537 5647 feature_gate.go:386] unrecognized feature gate: Example Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975540 5647 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975543 5647 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975546 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975549 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975552 5647 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975555 5647 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975557 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975560 5647 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975562 5647 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 17:23:33.001271 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975565 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975568 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975570 5647 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975573 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975575 5647 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975578 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975581 5647 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975584 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975586 5647 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975589 5647 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975592 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975595 5647 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975636 5647 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975640 5647 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975642 5647 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975645 5647 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975648 5647 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975650 5647 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975653 5647 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975655 5647 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 17:23:33.001923 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975658 5647 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975660 5647 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975663 5647 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975665 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975668 5647 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975671 5647 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975673 5647 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975676 5647 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975678 5647 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975681 5647 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975683 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975686 5647 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975689 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975691 5647 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975693 5647 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975697 5647 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975699 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975703 5647 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975707 5647 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 17:23:33.002496 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975710 5647 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975713 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975715 5647 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975718 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975721 5647 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975724 5647 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975726 5647 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975809 5647 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975814 5647 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975818 5647 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975822 5647 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975826 5647 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975830 5647 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:32.975834 5647 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.975921 5647 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.977279 5647 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.977355 5647 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.977577 5647 server.go:1014] "Starting client certificate rotation" Nov 02 17:23:33.003005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.978277 5647 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.978714 5647 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.978806 5647 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:17:20 +0000 UTC" deadline="2028-06-07 21:11:08.231224239 +0000 UTC" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.978858 5647 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13995h47m35.252369728s" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.979795 5647 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.981330 5647 log.go:25] "Validated CRI v1 runtime API" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.983707 5647 log.go:25] "Validated CRI v1 image API" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.984577 5647 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 17:23:33.003454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.987483 5647 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ada1dd5-3247-49ef-91de-acec385de42d:/dev/nvme0n1p4 bc8d1faa-9c72-4dd2-bd08-a619e9972794:/dev/nvme0n1p3] Nov 02 17:23:33.003842 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.987510 5647 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0/userdata/shm major:0 minor:385 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94acfe4437d84c18f686cc30b375026f945a6a93da63a79b6ebb88def859a6bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94acfe4437d84c18f686cc30b375026f945a6a93da63a79b6ebb88def859a6bb/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d/userdata/shm major:0 minor:317 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c13dd52c17fff47bebf980e36af54c6cae71de5eb4acdbb0997f8b93bb343668/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c13dd52c17fff47bebf980e36af54c6cae71de5eb4acdbb0997f8b93bb343668/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb213337ded4fd98e6ee65e89cdb24c47d0accbd42f5bf13b8288c73eff9ffe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb213337ded4fd98e6ee65e89cdb24c47d0accbd42f5bf13b8288c73eff9ffe3/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52/userdata/shm major:0 minor:399 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc/userdata/shm major:0 minor:386 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa/userdata/shm major:0 minor:314 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/019edf2b-0578-4009-8558-fdf6eb87d58d/volumes/kubernetes.io~projected/kube-api-access-rdmfb:{mountpoint:/var/lib/kubelet/pods/019edf2b-0578-4009-8558-fdf6eb87d58d/volumes/kubernetes.io~projected/kube-api-access-rdmfb major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/019edf2b-0578-4009-8558-fdf6eb87d58d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/019edf2b-0578-4009-8558-fdf6eb87d58d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c38f276-7bda-4a9d-a2b9-3e07385ed119/volumes/kubernetes.io~projected/kube-api-access-k5qfg:{mountpoint:/var/lib/kubelet/pods/0c38f276-7bda-4a9d-a2b9-3e07385ed119/volumes/kubernetes.io~projected/kube-api-access-k5qfg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~projected/kube-api-access-4fsz8:{mountpoint:/var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~projected/kube-api-access-4fsz8 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~projected/kube-api-access-m829p:{mountpoint:/var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~projected/kube-api-access-m829p major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45f8e074-1aca-444b-91e3-5f5bc339164f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/45f8e074-1aca-444b-91e3-5f5bc339164f/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b463fd2-4b01-4441-8d41-dbbe0a440a22/volumes/kubernetes.io~projected/kube-api-access-ws42m:{mountpoint:/var/lib/kubelet/pods/4b463fd2-4b01-4441-8d41-dbbe0a440a22/volumes/kubernetes.io~projected/kube-api-access-ws42m major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volumes/kubernetes.io~projected/kube-api-access-rxzmp:{mountpoint:/var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volumes/kubernetes.io~projected/kube-api-access-rxzmp major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6627b470-4507-4754-8d06-ef654308f832/volumes/kubernetes.io~projected/kube-api-access-rv8lq:{mountpoint:/var/lib/kubelet/pods/6627b470-4507-4754-8d06-ef654308f832/volumes/kubernetes.io~projected/kube-api-access-rv8lq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6627b470-4507-4754-8d06-ef654308f832/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/6627b470-4507-4754-8d06-ef654308f832/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~projected/kube-api-access-5vk5g:{mountpoint:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~projected/kube-api-access-5vk5g major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9021558c-9c68-4634-a371-3172c24a2d85/volumes/kubernetes.io~projected/kube-api-access-khkqp:{mountpoint:/var/lib/kubelet/pods/9021558c-9c68-4634-a371-3172c24a2d85/volumes/kubernetes.io~projected/kube-api-access-khkqp major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~projected/kube-api-access-mrj4d:{mountpoint:/var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~projected/kube-api-access-mrj4d major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf40ad1-2c0b-4b84-9242-4108e40de77f/volumes/kubernetes.io~projected/kube-api-access-cqgf5:{mountpoint:/var/lib/kubelet/pods/9cf40ad1-2c0b-4b84-9242-4108e40de77f/volumes/kubernetes.io~projected/kube-api-access-cqgf5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a/volumes/kubernetes.io~projected/kube-api-access-lq2nz:{mountpoint:/var/lib/kubelet/pods/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a/volumes/kubernetes.io~projected/kube-api-access-lq2nz major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/volumes/kubernetes.io~projected/kube-api-access-8rvb2:{mountpoint:/var/lib/kubelet/pods/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/volumes/kubernetes.io~projected/kube-api-access-8rvb2 major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b26856a8-af0c-48c4-b6a6-8042e9bea094/volumes/kubernetes.io~projected/kube-api-access-tjrcx:{mountpoint:/var/lib/kubelet/pods/b26856a8-af0c-48c4-b6a6-8042e9bea094/volumes/kubernetes.io~projected/kube-api-access-tjrcx major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd31aff-608d-4200-b76d-61f4c43e8a99/volumes/kubernetes.io~projected/kube-api-access-976gb:{mountpoint:/var/lib/kubelet/pods/bcd31aff-608d-4200-b76d-61f4c43e8a99/volumes/kubernetes.io~projected/kube-api-access-976gb major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcd31aff-608d-4200-b76d-61f4c43e8a99/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bcd31aff-608d-4200-b76d-61f4c43e8a99/volumes/kubernetes.io~secret/metrics-certs major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113/volumes/kubernetes.io~secret/tls-certificates major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10332e6-3673-471f-bd01-99c7b5d5ac3e/volumes/kubernetes.io~projected/kube-api-access-vq65c:{mountpoint:/var/lib/kubelet/pods/d10332e6-3673-471f-bd01-99c7b5d5ac3e/volumes/kubernetes.io~projected/kube-api-access-vq65c major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d10332e6-3673-471f-bd01-99c7b5d5ac3e/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d10332e6-3673-471f-bd01-99c7b5d5ac3e/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~projected/kube-api-access-fxhzj:{mountpoint:/var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~projected/kube-api-access-fxhzj major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8a199fd-2c80-44c0-9930-19820009ada9/volumes/kubernetes.io~projected/kube-api-access-949dj:{mountpoint:/var/lib/kubelet/pods/d8a199fd-2c80-44c0-9930-19820009ada9/volumes/kubernetes.io~projected/kube-api-access-949dj major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/alertmanager/9 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/config-reloader/6 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~empty-dir/config-out major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~projected/kube-api-access-n5cqw:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~projected/kube-api-access-n5cqw major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~projected/tls-assets major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/config-volume major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/web-config major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~projected/kube-api-access-khgsw:{mountpoint:/var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~projected/kube-api-access-khgsw major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed54c86c-1503-4c0d-b0b7-0454818c844c/volumes/kubernetes.io~projected/kube-api-access-stl27:{mountpoint:/var/lib/kubelet/pods/ed54c86c-1503-4c0d-b0b7-0454818c844c/volumes/kubernetes.io~projected/kube-api-access-stl27 major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4aa5bab490d7097e184508c922f52b985ec5b030141ab4d8ff46158e8648c9c4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/38acda02b7ca5f97101d4e05117e0385055e80179c4b17035caf9de81dd99496/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/acadf6a77acf2c8ff1cbb6f29e2e16815bc52f182bd46fc1aaaf5e050f7bcf75/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7f549f92ad9e52e14482b122ece210ca990daa9b70ec51072ebc9cc9f5faffcd/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/34bf9a50891f8fc23bbf292a34e2d4043fab7f89ce96c80daf2c30329b53409f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cba8b15082da3569e94e51adf2761379accd1411ccf924799079d3987c44b90f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/eca5c85cd805011b9a3e495a8e068f031f88a6932e7653b304c366ae088d4f7b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0cc7161eadac6f52090cd3115a2200054fff86174de6d534e8a0a9f628a59534/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/59447e7917b46246cde9afb650d5de0aaaa62d16d86a549dbb853625b88959cd/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-130:{mountpoint:/var/lib/containers/storage/overlay/b4540170af1a22dcc6090a906bf86210822695404154b1fdc2f7df373b86ea94/merged major:0 minor:130 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/0befcc8fd769ace852b2ceac022767d4e2f210b6890ce65fb92ecd16384f10fc/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/22e1953042c9c6dfea03e0a4ec2ff6fbb043ea7b5500e4595ea5416c087d21d3/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b984cf0522a92f11c1b0f778af9e3450bbeae5d21f61f086bfbd9dc38e5a4127/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/c3c961df252db20a5077cefb46ca2110990893fec0465f554f5c69cd6e655016/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/67c536b7784f81255ea59dbb2a11268caaefc27362c6520c65f5af69c69f7653/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/580324b003ea54f5863c8d8ec2a66542aa69aff772fed7379c67c3c820afaaff/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/277a0aceb7b68071ddff1fc38257d7d0707411c1a3b7fb696e7862858b72ab83/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/ed33d6841f08043432b7eca55aec964c6046f27abaa696f8767f0411b3d57359/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/87fe4590b2c6104c6a8f91f57295778e97085bc525f2030152bda3b1b97ddcb0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c4936e97c652cfc570adff8e9ffa2166f830df3495a49a22ecb9407812ac016e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d8ec619f6cca2392b97953cf01b436a29bfb8015d229407a89d1fbefede5a8e4/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/6d9d5464f2aed2d3aaf9a703dcdee2bd360f58c3c1eeb8489c3425c7042fe1c0/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8bd7fdaab0f44f4be8c6fb687daa3de0495abc233869d5009c38112963a8d307/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/0599abfc8451984237a3ca68a390d7b3b0ea8c771d27fd78cf19a5a439e65dd1/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/09e5e595c18804cd946b0a375092be0a3f981c9c3805fbbff47b795aeea2dd8a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/91e2bf4a7e286b61992ddf43bf47720d1fab1a46daeaf69052362d56c00fe5aa/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/c20c336d30c86b4c0a84f1a9492cf312135c332ad48f7d08362c11fbe3082a37/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/10cc38c7ee732621dd68607a0c45208d07f8cf576e4f77da1d79de8f3cbcdb24/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/a44d8546db12ca115ada7a97ded5601f20e9da6c1548beca46d14da8fc8632ab/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/61721adde010d59cbca59b1c931bc1dda1e4ef40c98e2daf83132cd0a15f2f3e/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/a5101e061e9387171ef27e24654471d382cb68eaa2a66efed832092ef1c8649a/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/018df25f5d872b7a5f9a52a0e0f9a8686b06d7bef700fe7bc7d58e019d0c0395/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/2aed130584c82a377f2a3f5af7f4b2c1902950a7392e5de59724e0823224dd15/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/7548dcc1b33678a4e90a611f45ea9d33ceed728ca2cc7d2633dcd9060be7f6b8/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/2d31f12258a22e539074bfd575c2f32769c4a9739b242000ead4aa77bbf09f14/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/d5ec2ad80249e93dbb2174fa9ef86981443eb1cd26c0547a2818d4968cc59e42/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/14ef078a144b62169f4c8c4b91347d692e9ec3d5c8c58d07647a945d09b7918e/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/24ee3c04bb6592ab25409d6fe5ec5e7dc6048f90d860d90840ed99aea3867b2b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/f387e2288ff32bb3183ca3aa249e2793badf03992dd37ef0b5bc1e3771521362/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6b6551e3ad5bde5b00a28829b3a25a55b202f8a1942dcfad7e785c09247e902d/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/4625b61fc2d45f7f8996ce25b68b2973ea43c0801c422e0fad948da261fb202a/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/2083ad7a92877577b794c86efe1c230255f06a62155f8f80100a0ac2221a848c/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/623844bc772c8344d0072e0ed620658099f095084f7cd6ccdbe6ec2c8b0d0f22/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/efc948541ded3f69e2e52b9153c596f8eb2aa2bb21388366af03e936fec73bf7/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/49715e1480fce94c46c572694d3c7791b917d4f910c878282d00df57e50e349e/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/66cc7d2cf9598a2fc35760639263367b575f59751d756344a971c7a44acf78a0/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/8f880416699b406a0c11cdfdec8fcc94e540f160511ad0c8045422ef9225d8da/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/b561edb745b6a7bfb57cfa5e2c0101f9ef506d7c14d5c2f5f94e7437301ef195/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/2432e88b756bf37355335a06e807d43c0e43d3f4a1a4450da5158385a0eba885/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/9405d3f48ac163f2f085dedca82627f48b2c055d1d59f9ce1c00657d72fa1a80/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/79a4c8217fc73b0a5bb0b453efb2435cee8721956f17bccddf7563fca172dfd6/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/79936764904a7cd861ed61d7316b24259682c5417475b765c52d668da00598a1/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/857e054443a493b7af963e971a1357b3cf0afa675ffb75d09e30e886cc85758d/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/d2e4e70de39460170e5a52afb8da61e4fa90a0d8aa015661f36a66ff0e2f410a/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/6457b8948de3f514d6f9257644d73fe75baeac46999cb598d649215d95e43011/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/5a4905cefe9ad3e160186a62852cb2242ef046e541b42bbfc26c25ee814acdf2/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/676ff7ccff5faeb62468e4ab6761e4fdfae2d0d88b208e8d5af8ce6ad3902278/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/7933ab9e82ce44a91111a892d45c349c8fe1aea45c9755cdbe3dd9084ac95dad/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/741ffbf1b9df7ef94e0fbad1299eba88ec199e96773f198c394f8f78ea83e9a3/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/9a22233c8be96925972bb9d49c219dab8a986a14303c40dbd21abd1ddeb8d966/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/1919898c85f779377b7ebc297cda386003e9d402b8f4436b1b13f6a1e75de219/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/7c7c096c663126fb58086207e4caafa440c43dc1bc87a66b3fa17edfb33f899f/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/f007129d134087a2ee1272494a21ec36b9ecab5787873669d3e334773ddfdcc5/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/3b19faee95360c7678cea1a244e98c9ddc0d16e0fd1b9dae2cbf87c6ebe48fe6/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/052896b02a4ae59cc77fca60d34d9c90184fcc6b031861816708210ed4f31fd1/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/8c6c8e8987c1ef78b2bb1d64ae47003ac60375fb6c7b1501e93926e51896cfe9/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b4b324f63c8b9805b40600033610457951e98bc00fb8f879b2ad3af159e4ee82/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/157bae39c99e0b66f2f313ca8ce1eed6818796c3495b7562bef94ed3b8c69c5a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a36b9602c9b54ad5f5490c0f61fe772ded0f0936da5c3fb1390d4e3e0b9e3ad5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3e8f2bb76956ae75917f26750cc5f06f257824920ed9c2bc057becd2c48c4732/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/01aa969eca1844e3bd1c50d6350edb6cb49e25230eff38f6c08095c8c1771a34/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/ed5da7e019f6bb74e61658d18a502e4a3dbe7db02538f48626fcaa7a116ff279/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/0fca41b765d892b4c7073f42bf0eab42ea9e35e576dc102776a5a421a7d700df/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/85eefc73ba032d1ac07ec80297a3f8eb0a98477f78a37950cce76f1cf4ddffd3/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.997220 5647 manager.go:217] Machine: {Timestamp:2026-11-02 17:23:32.996726266 +0000 UTC m=+0.070253461 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100070 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e6e8a1fcdbf9060b2408b70efa9e7 SystemUUID:ec2e6e8a-1fcd-bf90-60b2-408b70efa9e7 BootID:1b55da2b-d252-4a19-acf6-7dabd7f06c77 Filesystems:[{Device:/var/lib/kubelet/pods/4b463fd2-4b01-4441-8d41-dbbe0a440a22/volumes/kubernetes.io~projected/kube-api-access-ws42m DeviceMajor:0 DeviceMinor:398 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf03a1d0430e20a751acc305c76f83c4aed54cd8737af52659a445abbca28af2/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd31aff-608d-4200-b76d-61f4c43e8a99/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:306 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d8a199fd-2c80-44c0-9930-19820009ada9/volumes/kubernetes.io~projected/kube-api-access-949dj DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb213337ded4fd98e6ee65e89cdb24c47d0accbd42f5bf13b8288c73eff9ffe3/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-130 DeviceMajor:0 DeviceMinor:130 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:325 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94acfe4437d84c18f686cc30b375026f945a6a93da63a79b6ebb88def859a6bb/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e6bad23e6ed11557fdffea0b99ef8eed9d3ac8e68243132a29db5903e5a6b92/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~projected/kube-api-access-fxhzj DeviceMajor:0 DeviceMinor:281 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcd31aff-608d-4200-b76d-61f4c43e8a99/volumes/kubernetes.io~projected/kube-api-access-976gb DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/019edf2b-0578-4009-8558-fdf6eb87d58d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6627b470-4507-4754-8d06-ef654308f832/volumes/kubernetes.io~projected/kube-api-access-rv8lq DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~projected/kube-api-access-4fsz8 DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9021558c-9c68-4634-a371-3172c24a2d85/volumes/kubernetes.io~projected/kube-api-access-khkqp DeviceMajor:0 DeviceMinor:313 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:337 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45f8e074-1aca-444b-91e3-5f5bc339164f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf40ad1-2c0b-4b84-9242-4108e40de77f/volumes/kubernetes.io~projected/kube-api-access-cqgf5 DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58d8840db04d080d8147791d25c795a0a45c2ddd12ce4867f0dabb86a7ce4dc9/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e87c17f6b66315b3d0dfaec7ce4b83f12fdbbc67498b3e24f610158d1dfe9a38/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:394 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6627b470-4507-4754-8d06-ef654308f832/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11f6b4455bbc5337fbf6dc1d5bb234cce3368f033115f5232f5dcfb8167a9c4e/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19c138051660a5bf3f36cd2b37cd461a08e88f053ea6c457864e46433ce19fff/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af80756e8970d6bc821cc32e4fafc3b689165fd062875b6d6c03406fa0fb679d/userdata/shm DeviceMajor:0 DeviceMinor:317 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b26856a8-af0c-48c4-b6a6-8042e9bea094/volumes/kubernetes.io~projected/kube-api-access-tjrcx DeviceMajor:0 DeviceMinor:54 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/019edf2b-0578-4009-8558-fdf6eb87d58d/volumes/kubernetes.io~projected/kube-api-access-rdmfb DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~projected/kube-api-access-5vk5g DeviceMajor:0 DeviceMinor:384 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51ed35febf1b390140dd21db1cd5b96009800ae5ac12760c6452ac8b70a21fe0/userdata/shm DeviceMajor:0 DeviceMinor:385 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:257 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d1c34fcaeff9331ddddcbd24733172179e4a1ab15545dc4c790868f351323b1/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c668e39b76c686fc39993c89a6f07802f741a45148b8de01fb94dff6517740f/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9337c32d70dc0eb8e72287d77a7079dfccafb6bef53907597cef170e3cb9c6a1/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volumes/kubernetes.io~projected/kube-api-access-rxzmp DeviceMajor:0 DeviceMinor:51 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63987b50-f8b5-42a2-a5cf-50fc6cec635e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0fec2e842857eb1c5242042bb956446770200cc4e2305684ae1dedd536b08ecb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:333 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d72f4cc48ebb42288ef2717a18a38cca3d8c53ff738d780e0ee3d6481f70ba52/userdata/shm DeviceMajor:0 DeviceMinor:399 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~projected/kube-api-access-khgsw DeviceMajor:0 DeviceMinor:53 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0c38f276-7bda-4a9d-a2b9-3e07385ed119/volumes/kubernetes.io~projected/kube-api-access-k5qfg DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2496795bd86466ef3d2696110de011f3474050677d91935fc22f2609c437f70f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10332e6-3673-471f-bd01-99c7b5d5ac3e/volumes/kubernetes.io~projected/kube-api-access-vq65c DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/volumes/kubernetes.io~projected/kube-api-access-8rvb2 DeviceMajor:0 DeviceMinor:383 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:388 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6541049-99e9-453c-9953-8817f491dc4d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d10332e6-3673-471f-bd01-99c7b5d5ac3e/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb4d4a1273fef816aaec09f213104f4dc323f52539afe1682cdf902932db573e/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d15153aa-78ce-426a-9b63-5a4489d8fc45/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/volumes/kubernetes.io~projected/kube-api-access-mrj4d DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:332 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~projected/kube-api-access-n5cqw DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2158692e05f109b6683d7f525da66e84297897e3513c98bb74e9f8e058eb41d6/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/361abe17-e8b7-4695-ae30-8cd0ccc06017/volumes/kubernetes.io~projected/kube-api-access-m829p DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:379 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34b5f718-d276-44f2-b1b5-f9bf961147ed/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c13dd52c17fff47bebf980e36af54c6cae71de5eb4acdbb0997f8b93bb343668/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4645999d621c367af42a02687683761563472255e87ba94e3c9d09ecaa7a1fa/userdata/shm DeviceMajor:0 DeviceMinor:314 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a/volumes/kubernetes.io~projected/kube-api-access-lq2nz DeviceMajor:0 DeviceMinor:52 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42e371a5f6039cbb9cef9329ed610124003f9d101bb4a826e938dd17984f4c10/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dacaad1b9afd7b484706408618789484f0534fa338368e8b6ba6eab0ea9fa5fc/userdata/shm DeviceMajor:0 DeviceMinor:386 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/809d0903-74ab-4c73-a6db-5811776a58ec/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:393 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/ed54c86c-1503-4c0d-b0b7-0454818c844c/volumes/kubernetes.io~projected/kube-api-access-stl27 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53a6b61ce29d5cba710014173bd93f5a6727f21d872ee31d3cafae403ae249bd/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:312 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:336 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:42e371a5f6039cb MacAddress:6a:88:49:72:84:cb Speed:10000 Mtu:8901} {Name:51ed35febf1b390 MacAddress:0a:7d:9b:7c:89:73 Speed:10000 Mtu:8901} {Name:53a6b61ce29d5cb MacAddress:da:4c:12:50:52:0a Speed:10000 Mtu:8901} {Name:5d1c34fcaeff933 MacAddress:e2:ad:31:95:25:51 Speed:10000 Mtu:8901} {Name:5e6bad23e6ed115 MacAddress:4a:75:71:fa:44:f0 Speed:10000 Mtu:8901} {Name:9337c32d70dc0eb MacAddress:82:3f:36:66:cc:b1 Speed:10000 Mtu:8901} {Name:94acfe4437d84c1 MacAddress:ca:9d:60:8a:d4:da Speed:10000 Mtu:8901} {Name:af80756e8970d6b MacAddress:36:8c:11:6f:e5:e3 Speed:10000 Mtu:8901} {Name:bf03a1d0430e20a MacAddress:ee:a2:46:56:9e:26 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:3c:97:a2:5a:67 Speed:0 Mtu:9001} {Name:br-int MacAddress:36:30:dd:2b:bb:6d Speed:0 Mtu:8901} {Name:cb213337ded4fd9 MacAddress:52:41:ce:aa:16:99 Speed:10000 Mtu:8901} {Name:cb4d4a1273fef81 MacAddress:46:7c:f7:3a:2f:5c Speed:10000 Mtu:8901} {Name:d72f4cc48ebb422 MacAddress:9e:a3:a3:84:56:83 Speed:10000 Mtu:8901} {Name:dacaad1b9afd7b4 MacAddress:9e:d5:6b:1a:bd:26 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:3c:97:a2:5a:67 Speed:0 Mtu:9001} {Name:f4645999d621c36 MacAddress:1a:c8:65:6c:cf:b5 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:82:04:b5:af:1a:3a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:f2:b0:dd:7b:2d:e4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.997643 5647 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.997692 5647 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998007 5647 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998028 5647 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-71.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998164 5647 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998172 5647 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998189 5647 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998203 5647 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998325 5647 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998527 5647 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998644 5647 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998679 5647 kubelet.go:493] "Attempting to sync node with API server" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998689 5647 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998703 5647 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998713 5647 kubelet.go:405] "Adding apiserver pod source" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.998726 5647 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.999704 5647 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:32.999718 5647 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:23:33.004333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.000519 5647 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 17:23:33.018355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.018326 5647 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 17:23:33.018355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.018353 5647 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 17:23:33.018520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.018507 5647 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 17:23:33.018859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.018842 5647 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 17:23:33.018917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.018891 5647 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 17:23:33.019050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019038 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 17:23:33.019081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019062 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 17:23:33.019081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019073 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 17:23:33.019130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019081 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 17:23:33.019130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019091 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 17:23:33.019130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019101 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 17:23:33.019130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019111 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 17:23:33.019130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019121 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 17:23:33.019311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019132 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 17:23:33.019311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019142 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 17:23:33.019311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019162 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 17:23:33.019311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019177 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 17:23:33.019311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019216 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 17:23:33.019311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.019226 5647 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 17:23:33.023440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.023417 5647 server.go:1263] "Started kubelet" Nov 02 17:23:33.023574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.023502 5647 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 17:23:33.023719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.023667 5647 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 17:23:33.023831 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.023811 5647 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 17:23:33.024008 ip-10-0-128-71 systemd[1]: Started Kubernetes Kubelet. Nov 02 17:23:33.029013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.026648 5647 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 17:23:33.029013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.027380 5647 server.go:317] "Adding debug handlers to kubelet server" Nov 02 17:23:33.031781 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.031710 5647 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 17:23:33.032019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.031984 5647 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 17:23:33.032130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.032047 5647 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:17:21 +0000 UTC" deadline="2028-07-06 19:22:21.934110744 +0000 UTC" Nov 02 17:23:33.032130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.032066 5647 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14689h58m48.902047309s" Nov 02 17:23:33.032233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.032212 5647 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 17:23:33.032233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.032220 5647 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 17:23:33.032333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.032320 5647 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 17:23:33.035074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035022 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b5f718-d276-44f2-b1b5-f9bf961147ed" volumeName="kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-tls" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035076 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b5f718-d276-44f2-b1b5-f9bf961147ed" volumeName="kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035090 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361abe17-e8b7-4695-ae30-8cd0ccc06017" volumeName="kubernetes.io/projected/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-api-access-m829p" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035103 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b463fd2-4b01-4441-8d41-dbbe0a440a22" volumeName="kubernetes.io/empty-dir/4b463fd2-4b01-4441-8d41-dbbe0a440a22-tmp" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035115 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b463fd2-4b01-4441-8d41-dbbe0a440a22" volumeName="kubernetes.io/projected/4b463fd2-4b01-4441-8d41-dbbe0a440a22-kube-api-access-ws42m" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035126 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035142 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" volumeName="kubernetes.io/empty-dir/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-textfile" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035156 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd" volumeName="kubernetes.io/projected/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-kube-api-access-8rvb2" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035171 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="019edf2b-0578-4009-8558-fdf6eb87d58d" volumeName="kubernetes.io/projected/019edf2b-0578-4009-8558-fdf6eb87d58d-kube-api-access-rdmfb" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035184 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" volumeName="kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-tls" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035196 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c31ce7db-987c-4f05-9a4b-fbfd2fa0b113" volumeName="kubernetes.io/secret/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113-tls-certificates" seLinuxMountContext="" Nov 02 17:23:33.035212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035208 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035221 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035235 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.035234 5647 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035251 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6541049-99e9-453c-9953-8817f491dc4d" volumeName="kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-tmp" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035269 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c38f276-7bda-4a9d-a2b9-3e07385ed119" volumeName="kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cni-binary-copy" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035291 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361abe17-e8b7-4695-ae30-8cd0ccc06017" volumeName="kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-metrics-client-ca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035311 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361abe17-e8b7-4695-ae30-8cd0ccc06017" volumeName="kubernetes.io/empty-dir/361abe17-e8b7-4695-ae30-8cd0ccc06017-volume-directive-shadow" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035332 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d15153aa-78ce-426a-9b63-5a4489d8fc45" volumeName="kubernetes.io/projected/d15153aa-78ce-426a-9b63-5a4489d8fc45-kube-api-access-fxhzj" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035352 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6627b470-4507-4754-8d06-ef654308f832" volumeName="kubernetes.io/configmap/6627b470-4507-4754-8d06-ef654308f832-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035370 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6627b470-4507-4754-8d06-ef654308f832" volumeName="kubernetes.io/projected/6627b470-4507-4754-8d06-ef654308f832-kube-api-access-rv8lq" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035388 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" volumeName="kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035408 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" volumeName="kubernetes.io/projected/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-kube-api-access-lq2nz" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035426 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b26856a8-af0c-48c4-b6a6-8042e9bea094" volumeName="kubernetes.io/projected/b26856a8-af0c-48c4-b6a6-8042e9bea094-kube-api-access-tjrcx" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035472 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035484 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6627b470-4507-4754-8d06-ef654308f832" volumeName="kubernetes.io/empty-dir/6627b470-4507-4754-8d06-ef654308f832-data-volume" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035505 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-metrics-server-audit-profiles" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035518 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035535 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361abe17-e8b7-4695-ae30-8cd0ccc06017" volumeName="kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-tls" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035548 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-tls" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035560 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035574 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63987b50-f8b5-42a2-a5cf-50fc6cec635e" volumeName="kubernetes.io/projected/63987b50-f8b5-42a2-a5cf-50fc6cec635e-kube-api-access-rxzmp" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035586 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" volumeName="kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035598 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10332e6-3673-471f-bd01-99c7b5d5ac3e" volumeName="kubernetes.io/configmap/d10332e6-3673-471f-bd01-99c7b5d5ac3e-config-volume" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035611 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035623 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63987b50-f8b5-42a2-a5cf-50fc6cec635e" volumeName="kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035636 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" volumeName="kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035648 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf40ad1-2c0b-4b84-9242-4108e40de77f" volumeName="kubernetes.io/configmap/9cf40ad1-2c0b-4b84-9242-4108e40de77f-iptables-alerter-script" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035661 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" volumeName="kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-binary-copy" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035673 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6541049-99e9-453c-9953-8817f491dc4d" volumeName="kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-etc-tuned" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035686 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e" volumeName="kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035698 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b5f718-d276-44f2-b1b5-f9bf961147ed" volumeName="kubernetes.io/projected/34b5f718-d276-44f2-b1b5-f9bf961147ed-kube-api-access-4fsz8" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035712 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b26856a8-af0c-48c4-b6a6-8042e9bea094" volumeName="kubernetes.io/configmap/b26856a8-af0c-48c4-b6a6-8042e9bea094-serviceca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035725 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd31aff-608d-4200-b76d-61f4c43e8a99" volumeName="kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035756 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d15153aa-78ce-426a-9b63-5a4489d8fc45" volumeName="kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035778 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8a199fd-2c80-44c0-9930-19820009ada9" volumeName="kubernetes.io/projected/d8a199fd-2c80-44c0-9930-19820009ada9-kube-api-access-949dj" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035790 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="019edf2b-0578-4009-8558-fdf6eb87d58d" volumeName="kubernetes.io/secret/019edf2b-0578-4009-8558-fdf6eb87d58d-cert" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035805 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c38f276-7bda-4a9d-a2b9-3e07385ed119" volumeName="kubernetes.io/projected/0c38f276-7bda-4a9d-a2b9-3e07385ed119-kube-api-access-k5qfg" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035818 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035831 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d15153aa-78ce-426a-9b63-5a4489d8fc45" volumeName="kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035845 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035856 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6541049-99e9-453c-9953-8817f491dc4d" volumeName="kubernetes.io/projected/e6541049-99e9-453c-9953-8817f491dc4d-kube-api-access-khgsw" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035870 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-client-ca-bundle" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035884 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" volumeName="kubernetes.io/projected/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-kube-api-access-mrj4d" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035898 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd" volumeName="kubernetes.io/secret/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-monitoring-plugin-cert" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035910 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361abe17-e8b7-4695-ae30-8cd0ccc06017" volumeName="kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035939 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63987b50-f8b5-42a2-a5cf-50fc6cec635e" volumeName="kubernetes.io/secret/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035952 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9021558c-9c68-4634-a371-3172c24a2d85" volumeName="kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035963 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf40ad1-2c0b-4b84-9242-4108e40de77f" volumeName="kubernetes.io/projected/9cf40ad1-2c0b-4b84-9242-4108e40de77f-kube-api-access-cqgf5" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035975 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed54c86c-1503-4c0d-b0b7-0454818c844c" volumeName="kubernetes.io/projected/ed54c86c-1503-4c0d-b0b7-0454818c844c-kube-api-access-stl27" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035987 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6627b470-4507-4754-8d06-ef654308f832" volumeName="kubernetes.io/secret/6627b470-4507-4754-8d06-ef654308f832-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.035999 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34b5f718-d276-44f2-b1b5-f9bf961147ed" volumeName="kubernetes.io/configmap/34b5f718-d276-44f2-b1b5-f9bf961147ed-metrics-client-ca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036012 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45f8e074-1aca-444b-91e3-5f5bc339164f" volumeName="kubernetes.io/configmap/45f8e074-1aca-444b-91e3-5f5bc339164f-konnectivity-ca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036024 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/projected/809d0903-74ab-4c73-a6db-5811776a58ec-kube-api-access-5vk5g" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036037 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10332e6-3673-471f-bd01-99c7b5d5ac3e" volumeName="kubernetes.io/empty-dir/d10332e6-3673-471f-bd01-99c7b5d5ac3e-tmp-dir" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036049 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d15153aa-78ce-426a-9b63-5a4489d8fc45" volumeName="kubernetes.io/configmap/d15153aa-78ce-426a-9b63-5a4489d8fc45-metrics-client-ca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036065 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036077 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036095 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c38f276-7bda-4a9d-a2b9-3e07385ed119" volumeName="kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-daemon-config" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036107 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="361abe17-e8b7-4695-ae30-8cd0ccc06017" volumeName="kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036119 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63987b50-f8b5-42a2-a5cf-50fc6cec635e" volumeName="kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-script-lib" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036131 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10332e6-3673-471f-bd01-99c7b5d5ac3e" volumeName="kubernetes.io/projected/d10332e6-3673-471f-bd01-99c7b5d5ac3e-kube-api-access-vq65c" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036145 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036157 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45f8e074-1aca-444b-91e3-5f5bc339164f" volumeName="kubernetes.io/secret/45f8e074-1aca-444b-91e3-5f5bc339164f-agent-certs" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036170 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63987b50-f8b5-42a2-a5cf-50fc6cec635e" volumeName="kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-env-overrides" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036183 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="809d0903-74ab-4c73-a6db-5811776a58ec" volumeName="kubernetes.io/empty-dir/809d0903-74ab-4c73-a6db-5811776a58ec-audit-log" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036197 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" volumeName="kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-metrics-client-ca" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036210 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" volumeName="kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036223 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036245 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e64a1f95-269d-4654-bef7-e9ba232ce9ee" volumeName="kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036267 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bcd31aff-608d-4200-b76d-61f4c43e8a99" volumeName="kubernetes.io/projected/bcd31aff-608d-4200-b76d-61f4c43e8a99-kube-api-access-976gb" seLinuxMountContext="" Nov 02 17:23:33.036234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036289 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d10332e6-3673-471f-bd01-99c7b5d5ac3e" volumeName="kubernetes.io/secret/d10332e6-3673-471f-bd01-99c7b5d5ac3e-metrics-tls" seLinuxMountContext="" Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036311 5647 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed54c86c-1503-4c0d-b0b7-0454818c844c" volumeName="kubernetes.io/empty-dir/ed54c86c-1503-4c0d-b0b7-0454818c844c-tmp-dir" seLinuxMountContext="" Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036327 5647 reconstruct.go:97] "Volume reconstruction finished" Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.036342 5647 reconciler.go:29] "Reconciler: start to sync state" Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037238 5647 factory.go:55] Registering systemd factory Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037259 5647 factory.go:223] Registration of the systemd container factory successfully Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037648 5647 factory.go:153] Registering CRI-O factory Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037668 5647 factory.go:223] Registration of the crio container factory successfully Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037750 5647 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037774 5647 factory.go:103] Registering Raw factory Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.037793 5647 manager.go:1199] Started watching for new ooms in manager Nov 02 17:23:33.041068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.038138 5647 manager.go:319] Starting recovery of all containers Nov 02 17:23:33.071699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.069405 5647 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 17:23:33.073100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.073076 5647 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 17:23:33.073100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.073100 5647 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 17:23:33.073229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.073120 5647 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 17:23:33.073229 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.073158 5647 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 17:23:33.099124 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.099080 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3" exitCode=0 Nov 02 17:23:33.100764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.100717 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:23:33.105256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.105237 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:23:33.135406 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.135381 5647 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" Nov 02 17:23:33.135638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.135623 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:23:33.136027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.135997 5647 generic.go:356] "Generic (PLEG): container finished" podID="63987b50-f8b5-42a2-a5cf-50fc6cec635e" containerID="3d8a40a4f2207b7b616302dfd0b688f873000ddd0a8482c846d8ed5431aae783" exitCode=1 Nov 02 17:23:33.144933 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.144905 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="dd6cc80c8146f12979d66be020d15a7d80c0c7504000cee12f800553aab921fa" exitCode=0 Nov 02 17:23:33.144933 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.144928 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="735dac0c9905990e3900e28438595fbc95aa4fc8324500b72ff7faca68c46189" exitCode=0 Nov 02 17:23:33.145060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.144938 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="3711ebff2e38eddfc07413aee7f7913d14c6e16541f125aeec6541b7d1807eb3" exitCode=0 Nov 02 17:23:33.145060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.144948 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="14a40833d54f3c46177c99e584babd8bb9a2f878521278fad73b112f6d286a03" exitCode=0 Nov 02 17:23:33.145060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.144956 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="a57276a1bf7264356604c9d47cefb7899645cb6417c7a9dd59cdbd76219276a6" exitCode=0 Nov 02 17:23:33.145060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.144964 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f458e79-ca1c-4eb2-9ddd-f3daacf6055a" containerID="a9e9c513df1a3b12779a4ccb89a0835b8f72eb823ec57d07fa9d5b2bd516e644" exitCode=0 Nov 02 17:23:33.151702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.151676 5647 generic.go:356] "Generic (PLEG): container finished" podID="9a23d348-6b2d-4ded-9ff9-543a1e01f6cf" containerID="43acd66ff34fbb30c51c6a47fe2374eb2ed63671ec28d2625f00d3cdebe2ca82" exitCode=0 Nov 02 17:23:33.166459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.166418 5647 generic.go:356] "Generic (PLEG): container finished" podID="a94799fcc07ccf47af40bea71d377682" containerID="d7743a49a497c925f3daed0eb78afa202f453aa245a4ed622af3801f6500928e" exitCode=0 Nov 02 17:23:33.173254 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.173224 5647 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 17:23:33.174704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.174687 5647 manager.go:324] Recovery completed Nov 02 17:23:33.198037 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.198013 5647 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:23:33.200195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.200179 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:23:33.200284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.200210 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:23:33.200284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.200220 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:23:33.203539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203520 5647 cpu_manager.go:226] "Starting" policy="none" Nov 02 17:23:33.203539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203536 5647 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 17:23:33.203659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203557 5647 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 17:23:33.203659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203539 5647 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:23:33.203775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203685 5647 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 17:23:33.203775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203695 5647 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 17:23:33.203775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203714 5647 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 17:23:33.203775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203720 5647 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 17:23:33.203775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203725 5647 policy_none.go:50] "Start" Nov 02 17:23:33.203775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203770 5647 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 17:23:33.203998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203780 5647 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 17:23:33.203998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203903 5647 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 17:23:33.203998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203910 5647 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 17:23:33.203998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.203915 5647 policy_none.go:44] "Start" Nov 02 17:23:33.204612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.204597 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:23:33.204679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.204630 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:23:33.204679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.204658 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:23:33.210722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.210015 5647 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 17:23:33.213918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.213902 5647 manager.go:341] "Starting Device Plugin manager" Nov 02 17:23:33.213989 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.213930 5647 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 17:23:33.213989 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.213939 5647 server.go:87] "Starting device plugin registration server" Nov 02 17:23:33.214282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214269 5647 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 17:23:33.214320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214285 5647 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214376 5647 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214487 5647 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214526 5647 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214534 5647 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.214628 5647 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:23:33.214516224Z","UUID":"d2cdaa48-d2e4-4c05-8f20-65b5f7d8530a","Handler":null,"Name":"","Endpoint":""} Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.216049 5647 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.216078 5647 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-71.us-west-1.compute.internal\" not found" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.216968 5647 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.217024 5647 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 17:23:33.219045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.217049 5647 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 17:23:33.223073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.219242 5647 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:23:33.223073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.219364 5647 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:23:33.223073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.219417 5647 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:23:33.314904 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.314861 5647 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 17:23:33.316371 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.316349 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 02 17:23:33.316482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.316383 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 02 17:23:33.316482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.316394 5647 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-71.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 02 17:23:33.316482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.316437 5647 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.316991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.316974 5647 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 17:23:33.373613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.373569 5647 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal"] Nov 02 17:23:33.374277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374252 5647 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.374277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374274 5647 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.374428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374256 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a8894bf200d7f2fbf8f4d4549108d68f","Type":"ContainerStarted","Data":"8ed51046e461248943a1f736a48cad9b9e86e78d10afc13db52a38a37ae80a2c"} Nov 02 17:23:33.374428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374312 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a8894bf200d7f2fbf8f4d4549108d68f","Type":"ContainerStarted","Data":"052bc6d2d6a862db58b63340864a19697f9752fd334e53e095bf49c9f19dfca8"} Nov 02 17:23:33.374428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374348 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a94799fcc07ccf47af40bea71d377682","Type":"ContainerStarted","Data":"f2a2af8c18445ef5bec69cc7c7e8677e5ff2a1a8f168995e53e6954680c52c93"} Nov 02 17:23:33.374428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374361 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a94799fcc07ccf47af40bea71d377682","Type":"ContainerDied","Data":"d7743a49a497c925f3daed0eb78afa202f453aa245a4ed622af3801f6500928e"} Nov 02 17:23:33.374428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.374375 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" event={"ID":"a94799fcc07ccf47af40bea71d377682","Type":"ContainerStarted","Data":"1968ad71668c9f3ec9a0ab6fc9607c430bdf98535005e5f80f027536bd9ca70f"} Nov 02 17:23:33.430638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.430605 5647 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.430803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.430677 5647 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.440465 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.440446 5647 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:23:33.440572 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.440497 5647 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.441297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.441278 5647 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:23:33.441345 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:33.441319 5647 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.518561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.518475 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.518561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.518524 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.518561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.518544 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8894bf200d7f2fbf8f4d4549108d68f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a8894bf200d7f2fbf8f4d4549108d68f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.619341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.619300 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.619341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.619343 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8894bf200d7f2fbf8f4d4549108d68f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a8894bf200d7f2fbf8f4d4549108d68f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.619540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.619365 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.619540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.619408 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.619540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.619409 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a94799fcc07ccf47af40bea71d377682-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a94799fcc07ccf47af40bea71d377682\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:33.619540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.619421 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8894bf200d7f2fbf8f4d4549108d68f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" (UID: \"a8894bf200d7f2fbf8f4d4549108d68f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:34.000025 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:33.999991 5647 apiserver.go:52] "Watching apiserver" Nov 02 17:23:34.135995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.135961 5647 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 17:23:34.136629 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.136590 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-dc65z","openshift-cluster-node-tuning-operator/tuned-dj5gx","openshift-monitoring/node-exporter-jmf49","kube-system/konnectivity-agent-x89xb","openshift-dns/dns-default-9sr25","openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq","kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal","openshift-monitoring/metrics-server-6f564b5ffb-5scd2","openshift-multus/multus-bwsqh","openshift-dns/node-resolver-2l4bx","openshift-image-registry/node-ca-w8qzs","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8","openshift-monitoring/monitoring-plugin-77f4f76594-gjj99","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p","openshift-multus/network-metrics-daemon-hh9rw","openshift-ovn-kubernetes/ovnkube-node-67tdm","openshift-console/downloads-6dd789c8b7-wwch5","openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97","openshift-insights/insights-runtime-extractor-kmkz5","openshift-network-diagnostics/network-check-target-tlgwq","openshift-network-operator/iptables-alerter-hq2f5","openshift-ingress-canary/ingress-canary-zg8xf","openshift-monitoring/alertmanager-main-0","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf","kube-system/global-pull-secret-syncer-pxq56"] Nov 02 17:23:34.158381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.158345 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.171107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.171086 5647 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:34.171293 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.171277 5647 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:34.191398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191370 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.191398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191372 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.191574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191439 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.191574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191504 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.191574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191527 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.191808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191772 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.191910 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.191859 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.194070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194056 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 17:23:34.194154 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194104 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 17:23:34.194218 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194162 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 17:23:34.194541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194529 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 17:23:34.194609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194576 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 17:23:34.194645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194609 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.194837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194820 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.194945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194843 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.194945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.194865 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 17:23:34.195387 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.195372 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.195447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.195411 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 17:23:34.195613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.195598 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.197834 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.197816 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 17:23:34.230894 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.230867 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-v5b9c\"" Nov 02 17:23:34.231136 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231119 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkkxw\"" Nov 02 17:23:34.231194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231184 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 17:23:34.231395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231382 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 17:23:34.231435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231426 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j6nt8\"" Nov 02 17:23:34.231568 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231558 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7kdpj\"" Nov 02 17:23:34.231647 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231636 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x4464\"" Nov 02 17:23:34.231721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231711 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-867zk\"" Nov 02 17:23:34.231806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.231795 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 17:23:34.232276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232256 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l2rxq\"" Nov 02 17:23:34.232446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232336 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jxqfl\"" Nov 02 17:23:34.232446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232396 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9wf5x\"" Nov 02 17:23:34.232446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232439 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l2dvj\"" Nov 02 17:23:34.232540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232480 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 17:23:34.232724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232708 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-546w9\"" Nov 02 17:23:34.232836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232825 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 17:23:34.232907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232898 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2zcw\"" Nov 02 17:23:34.232985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.232976 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 17:23:34.233045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.233037 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ztdmt\"" Nov 02 17:23:34.238309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.238290 5647 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 17:23:34.238936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.238923 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-b2f84\"" Nov 02 17:23:34.239650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.239635 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 17:23:34.240008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.239983 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b28rw\"" Nov 02 17:23:34.240098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.240023 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 02 17:23:34.240098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.240057 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-clgvl\"" Nov 02 17:23:34.240098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.240078 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:23:34.240098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.240078 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 17:23:34.240267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.240251 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 17:23:34.240336 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.240316 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-svxv2\"" Nov 02 17:23:34.241658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.241644 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 02 17:23:34.241704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.241659 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 17:23:34.241888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.241873 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:23:34.241947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.241935 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-jbq9x\"" Nov 02 17:23:34.242956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.242914 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:23:34.243433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243413 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:23:34.243565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243441 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:23:34.243565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243447 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:23:34.243565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243451 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 02 17:23:34.243565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243460 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:23:34.243565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243505 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243590 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243595 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sc9pn\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243617 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ar746rs3icng6\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243647 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243657 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243659 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243674 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243670 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243619 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-z497l\"" Nov 02 17:23:34.243853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243720 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hmltn\"" Nov 02 17:23:34.244315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243939 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:23:34.244315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.243975 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:23:34.244315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.244100 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 17:23:34.244537 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.244519 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:23:34.244683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.244662 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8lnng\"" Nov 02 17:23:34.244818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.244798 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:23:34.245014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.244998 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hv5xv\"" Nov 02 17:23:34.245195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245177 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fvvhud3u970td\"" Nov 02 17:23:34.245397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245382 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:23:34.245522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.244676 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:23:34.245574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245545 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zwtpn\"" Nov 02 17:23:34.245671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245652 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9fqjw\"" Nov 02 17:23:34.245728 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245683 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 02 17:23:34.245728 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245701 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 17:23:34.245905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.245888 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 02 17:23:34.247591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.247566 5647 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:23:34.247695 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:34.247647 5647 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:34.248240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.248219 5647 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 17:23:34.248385 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:34.248281 5647 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-71.us-west-1.compute.internal" Nov 02 17:23:34.293410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293334 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 17:23:34.293587 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293568 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.293650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293634 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:23:34.293650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293644 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 02 17:23:34.293729 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293652 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.293729 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293658 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.293729 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293654 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:23:34.293945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293926 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.294078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293933 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293939 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293954 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293955 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294114 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293985 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293998 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.293999 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294081 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294146 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294180 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.294194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294187 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 17:23:34.294695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294312 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 02 17:23:34.294695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294344 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 17:23:34.294695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294608 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 17:23:34.294997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.294980 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 17:23:34.296440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.296418 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:23:34.296530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.296446 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:23:34.296569 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.296552 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:23:34.323785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323727 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-systemd-units\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.323893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323800 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6627b470-4507-4754-8d06-ef654308f832-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.323893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323858 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.323893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323886 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323904 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-kubelet\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323922 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cni-binary-copy\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.323939 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vk5g\" (UniqueName: \"kubernetes.io/projected/809d0903-74ab-4c73-a6db-5811776a58ec-kube-api-access-5vk5g\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324062 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrcx\" (UniqueName: \"kubernetes.io/projected/b26856a8-af0c-48c4-b6a6-8042e9bea094-kube-api-access-tjrcx\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324096 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cnibin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324111 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6627b470-4507-4754-8d06-ef654308f832-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.324141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324127 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-config\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.324423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324158 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-sys\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.324423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324194 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.324423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324203 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.324423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324234 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.324423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324280 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cni-binary-copy\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.324423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324347 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324436 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-config\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324454 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-run\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324495 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324510 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324571 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324611 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324650 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.324694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324693 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws42m\" (UniqueName: \"kubernetes.io/projected/4b463fd2-4b01-4441-8d41-dbbe0a440a22-kube-api-access-ws42m\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324715 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324778 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-system-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324795 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324805 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-conf-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324823 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-client-certs\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324865 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwmxf\" (UID: \"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324888 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.325053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.324982 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325077 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-node-log\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325075 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-client-certs\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325117 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325138 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c31ce7db-987c-4f05-9a4b-fbfd2fa0b113-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-lwmxf\" (UID: \"c31ce7db-987c-4f05-9a4b-fbfd2fa0b113\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325148 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgsw\" (UniqueName: \"kubernetes.io/projected/e6541049-99e9-453c-9953-8817f491dc4d-kube-api-access-khgsw\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325195 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15153aa-78ce-426a-9b63-5a4489d8fc45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325303 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325345 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.325379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325375 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-netd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325411 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgf5\" (UniqueName: \"kubernetes.io/projected/9cf40ad1-2c0b-4b84-9242-4108e40de77f-kube-api-access-cqgf5\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325433 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976gb\" (UniqueName: \"kubernetes.io/projected/bcd31aff-608d-4200-b76d-61f4c43e8a99-kube-api-access-976gb\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325449 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d15153aa-78ce-426a-9b63-5a4489d8fc45-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325457 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809d0903-74ab-4c73-a6db-5811776a58ec-audit-log\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325480 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325488 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325602 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325648 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325636 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/809d0903-74ab-4c73-a6db-5811776a58ec-audit-log\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-dbus\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325697 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325709 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325760 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.325797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325789 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325821 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-sys-fs\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325926 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-etc-kubernetes\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325928 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325972 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.325982 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-env-overrides\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326010 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-script-lib\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326032 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-conf\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326079 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-sys\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326120 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-textfile\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326166 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326204 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-textfile\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326206 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-env-overrides\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326245 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmfb\" (UniqueName: \"kubernetes.io/projected/019edf2b-0578-4009-8558-fdf6eb87d58d-kube-api-access-rdmfb\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326256 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovnkube-script-lib\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326278 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326304 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326328 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stl27\" (UniqueName: \"kubernetes.io/projected/ed54c86c-1503-4c0d-b0b7-0454818c844c-kube-api-access-stl27\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326352 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-socket-dir-parent\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326384 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.326399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326391 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-k8s-cni-cncf-io\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326572 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326637 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-wtmp\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326657 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj4d\" (UniqueName: \"kubernetes.io/projected/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-kube-api-access-mrj4d\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326673 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-multus\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326706 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326753 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-multus-certs\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326778 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-client-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326805 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326897 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326969 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45f8e074-1aca-444b-91e3-5f5bc339164f-konnectivity-ca\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.326991 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-client-ca-bundle\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.327064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327023 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b26856a8-af0c-48c4-b6a6-8042e9bea094-serviceca\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327113 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-bin\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327162 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/45f8e074-1aca-444b-91e3-5f5bc339164f-konnectivity-ca\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327182 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327199 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf40ad1-2c0b-4b84-9242-4108e40de77f-host-slash\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327252 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327295 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-system-cni-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327342 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmf4g\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327375 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949dj\" (UniqueName: \"kubernetes.io/projected/d8a199fd-2c80-44c0-9930-19820009ada9-kube-api-access-949dj\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327401 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-kubelet\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327428 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10332e6-3673-471f-bd01-99c7b5d5ac3e-tmp-dir\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327448 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327466 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd31aff-608d-4200-b76d-61f4c43e8a99-metrics-certs\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:34.327483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327475 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-modprobe-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327494 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf40ad1-2c0b-4b84-9242-4108e40de77f-iptables-alerter-script\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327534 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-metrics-server-audit-profiles\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327604 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d10332e6-3673-471f-bd01-99c7b5d5ac3e-tmp-dir\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327621 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327640 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b26856a8-af0c-48c4-b6a6-8042e9bea094-serviceca\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327648 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34b5f718-d276-44f2-b1b5-f9bf961147ed-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327655 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf40ad1-2c0b-4b84-9242-4108e40de77f-iptables-alerter-script\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327677 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-root\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327775 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-bin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327801 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-var-lib-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327827 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-etc-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327839 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327853 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-tls\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327871 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327890 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-netns\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327889 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/809d0903-74ab-4c73-a6db-5811776a58ec-metrics-server-audit-profiles\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327910 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10332e6-3673-471f-bd01-99c7b5d5ac3e-config-volume\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327948 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2nz\" (UniqueName: \"kubernetes.io/projected/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-kube-api-access-lq2nz\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.327974 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzmp\" (UniqueName: \"kubernetes.io/projected/63987b50-f8b5-42a2-a5cf-50fc6cec635e-kube-api-access-rxzmp\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328016 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-tls\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328026 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d10332e6-3673-471f-bd01-99c7b5d5ac3e-config-volume\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.328044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328026 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10332e6-3673-471f-bd01-99c7b5d5ac3e-metrics-tls\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328113 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328172 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-daemon-config\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328247 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d10332e6-3673-471f-bd01-99c7b5d5ac3e-metrics-tls\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328288 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328317 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-host\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328325 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328344 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019edf2b-0578-4009-8558-fdf6eb87d58d-cert\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328375 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed54c86c-1503-4c0d-b0b7-0454818c844c-tmp-dir\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328390 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-daemon-config\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328394 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-kubelet-config\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328431 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328457 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed54c86c-1503-4c0d-b0b7-0454818c844c-tmp-dir\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328463 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cnibin\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328489 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328521 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-socket-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328553 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/019edf2b-0578-4009-8558-fdf6eb87d58d-cert\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328566 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-tmp\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328604 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328634 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328646 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-registration-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328649 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.328670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328647 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-tmp\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328760 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-device-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328788 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-netns\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328810 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qfg\" (UniqueName: \"kubernetes.io/projected/0c38f276-7bda-4a9d-a2b9-3e07385ed119-kube-api-access-k5qfg\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328846 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-kubernetes\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328865 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-lib-modules\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328925 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv8lq\" (UniqueName: \"kubernetes.io/projected/6627b470-4507-4754-8d06-ef654308f832-kube-api-access-rv8lq\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.328979 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhzj\" (UniqueName: \"kubernetes.io/projected/d15153aa-78ce-426a-9b63-5a4489d8fc45-kube-api-access-fxhzj\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329061 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329080 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329124 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329152 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-slash\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329235 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6627b470-4507-4754-8d06-ef654308f832-crio-socket\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329252 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.329273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329264 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361abe17-e8b7-4695-ae30-8cd0ccc06017-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329303 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/361abe17-e8b7-4695-ae30-8cd0ccc06017-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329303 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329325 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b5f718-d276-44f2-b1b5-f9bf961147ed-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329347 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329374 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovn-node-metrics-cert\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329399 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-var-lib-kubelet\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329423 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627b470-4507-4754-8d06-ef654308f832-data-volume\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329447 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6627b470-4507-4754-8d06-ef654308f832-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329494 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-tls\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329507 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6627b470-4507-4754-8d06-ef654308f832-data-volume\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329521 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329542 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329552 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34b5f718-d276-44f2-b1b5-f9bf961147ed-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329546 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329600 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/63987b50-f8b5-42a2-a5cf-50fc6cec635e-ovn-node-metrics-cert\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329601 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-hostroot\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329628 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-etc-tuned\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329654 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6627b470-4507-4754-8d06-ef654308f832-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329661 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-metrics-client-ca\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329686 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329702 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.329754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329687 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e6541049-99e9-453c-9953-8817f491dc4d-etc-tuned\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329794 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/809d0903-74ab-4c73-a6db-5811776a58ec-secret-metrics-server-tls\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329810 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329830 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329845 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329850 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329902 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.329907 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-metrics-client-ca\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330041 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330066 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330086 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330087 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330117 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45f8e074-1aca-444b-91e3-5f5bc339164f-agent-certs\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330154 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-ovn\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330268 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq65c\" (UniqueName: \"kubernetes.io/projected/d10332e6-3673-471f-bd01-99c7b5d5ac3e-kube-api-access-vq65c\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330302 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/45f8e074-1aca-444b-91e3-5f5bc339164f-agent-certs\") pod \"konnectivity-agent-x89xb\" (UID: \"45f8e074-1aca-444b-91e3-5f5bc339164f\") " pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330298 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvb2\" (UniqueName: \"kubernetes.io/projected/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-kube-api-access-8rvb2\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330344 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330432 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330491 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330511 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330532 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330558 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysconfig\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330578 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-systemd\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330593 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-systemd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330608 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-log-socket\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330611 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-original-pull-secret\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330641 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330664 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m829p\" (UniqueName: \"kubernetes.io/projected/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-api-access-m829p\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.330755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330708 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b463fd2-4b01-4441-8d41-dbbe0a440a22-tmp\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330784 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b26856a8-af0c-48c4-b6a6-8042e9bea094-host\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330789 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b463fd2-4b01-4441-8d41-dbbe0a440a22-tmp\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330812 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330827 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330830 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cni-binary-copy\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330857 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330867 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330895 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsz8\" (UniqueName: \"kubernetes.io/projected/34b5f718-d276-44f2-b1b5-f9bf961147ed-kube-api-access-4fsz8\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.330965 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed54c86c-1503-4c0d-b0b7-0454818c844c-hosts-file\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.331028 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d15153aa-78ce-426a-9b63-5a4489d8fc45-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.331600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.331064 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.381785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.381726 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgsw\" (UniqueName: \"kubernetes.io/projected/e6541049-99e9-453c-9953-8817f491dc4d-kube-api-access-khgsw\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.381941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.381919 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws42m\" (UniqueName: \"kubernetes.io/projected/4b463fd2-4b01-4441-8d41-dbbe0a440a22-kube-api-access-ws42m\") pod \"downloads-6dd789c8b7-wwch5\" (UID: \"4b463fd2-4b01-4441-8d41-dbbe0a440a22\") " pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:34.382008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.381958 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrcx\" (UniqueName: \"kubernetes.io/projected/b26856a8-af0c-48c4-b6a6-8042e9bea094-kube-api-access-tjrcx\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.382278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.382258 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vk5g\" (UniqueName: \"kubernetes.io/projected/809d0903-74ab-4c73-a6db-5811776a58ec-kube-api-access-5vk5g\") pod \"metrics-server-6f564b5ffb-5scd2\" (UID: \"809d0903-74ab-4c73-a6db-5811776a58ec\") " pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:34.386064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.386046 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khkqp\" (UniqueName: \"kubernetes.io/projected/9021558c-9c68-4634-a371-3172c24a2d85-kube-api-access-khkqp\") pod \"network-check-target-tlgwq\" (UID: \"9021558c-9c68-4634-a371-3172c24a2d85\") " pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:34.387623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.387601 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976gb\" (UniqueName: \"kubernetes.io/projected/bcd31aff-608d-4200-b76d-61f4c43e8a99-kube-api-access-976gb\") pod \"network-metrics-daemon-hh9rw\" (UID: \"bcd31aff-608d-4200-b76d-61f4c43e8a99\") " pod="openshift-multus/network-metrics-daemon-hh9rw" Nov 02 17:23:34.387711 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.387695 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmfb\" (UniqueName: \"kubernetes.io/projected/019edf2b-0578-4009-8558-fdf6eb87d58d-kube-api-access-rdmfb\") pod \"ingress-canary-zg8xf\" (UID: \"019edf2b-0578-4009-8558-fdf6eb87d58d\") " pod="openshift-ingress-canary/ingress-canary-zg8xf" Nov 02 17:23:34.387848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.387833 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj4d\" (UniqueName: \"kubernetes.io/projected/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-kube-api-access-mrj4d\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.388553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.388536 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stl27\" (UniqueName: \"kubernetes.io/projected/ed54c86c-1503-4c0d-b0b7-0454818c844c-kube-api-access-stl27\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.389402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.389383 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgf5\" (UniqueName: \"kubernetes.io/projected/9cf40ad1-2c0b-4b84-9242-4108e40de77f-kube-api-access-cqgf5\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.391407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.391387 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") pod \"alertmanager-main-0\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:23:34.391996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.391959 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzmp\" (UniqueName: \"kubernetes.io/projected/63987b50-f8b5-42a2-a5cf-50fc6cec635e-kube-api-access-rxzmp\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.391996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.391979 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv8lq\" (UniqueName: \"kubernetes.io/projected/6627b470-4507-4754-8d06-ef654308f832-kube-api-access-rv8lq\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.392144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.392091 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fsz8\" (UniqueName: \"kubernetes.io/projected/34b5f718-d276-44f2-b1b5-f9bf961147ed-kube-api-access-4fsz8\") pod \"prometheus-operator-9cd89b74c-m2bk8\" (UID: \"34b5f718-d276-44f2-b1b5-f9bf961147ed\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-m2bk8" Nov 02 17:23:34.392264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.392244 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq65c\" (UniqueName: \"kubernetes.io/projected/d10332e6-3673-471f-bd01-99c7b5d5ac3e-kube-api-access-vq65c\") pod \"dns-default-9sr25\" (UID: \"d10332e6-3673-471f-bd01-99c7b5d5ac3e\") " pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:34.392301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.392247 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qfg\" (UniqueName: \"kubernetes.io/projected/0c38f276-7bda-4a9d-a2b9-3e07385ed119-kube-api-access-k5qfg\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.392598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.392575 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvb2\" (UniqueName: \"kubernetes.io/projected/a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd-kube-api-access-8rvb2\") pod \"monitoring-plugin-77f4f76594-gjj99\" (UID: \"a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:34.392706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.392691 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m829p\" (UniqueName: \"kubernetes.io/projected/361abe17-e8b7-4695-ae30-8cd0ccc06017-kube-api-access-m829p\") pod \"kube-state-metrics-5dc8d7d89b-7glpq\" (UID: \"361abe17-e8b7-4695-ae30-8cd0ccc06017\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7glpq" Nov 02 17:23:34.393041 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.393024 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2nz\" (UniqueName: \"kubernetes.io/projected/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-kube-api-access-lq2nz\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.393401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.393384 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhzj\" (UniqueName: \"kubernetes.io/projected/d15153aa-78ce-426a-9b63-5a4489d8fc45-kube-api-access-fxhzj\") pod \"openshift-state-metrics-7bd7489957-lbz97\" (UID: \"d15153aa-78ce-426a-9b63-5a4489d8fc45\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lbz97" Nov 02 17:23:34.393517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.393500 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-949dj\" (UniqueName: \"kubernetes.io/projected/d8a199fd-2c80-44c0-9930-19820009ada9-kube-api-access-949dj\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.431554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed54c86c-1503-4c0d-b0b7-0454818c844c-hosts-file\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.431672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431567 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.431672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431593 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-systemd-units\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.431672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431616 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.431672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431637 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-kubelet\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.431672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431661 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cnibin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.431943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431685 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-sys\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.431943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431712 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.431943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431832 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-run\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.431943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431870 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.431943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431901 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.431943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431934 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-system-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.432191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431962 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-conf-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.432191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.431993 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.432191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432052 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-run\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.432191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432093 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-kubelet\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432127 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-system-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.432400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432201 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-systemd-units\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432261 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-conf-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.432400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432351 5647 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 17:23:34.432400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432396 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-sys\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.432591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432441 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.432591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432543 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-cnibin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.432693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432582 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed54c86c-1503-4c0d-b0b7-0454818c844c-hosts-file\") pod \"node-resolver-2l4bx\" (UID: \"ed54c86c-1503-4c0d-b0b7-0454818c844c\") " pod="openshift-dns/node-resolver-2l4bx" Nov 02 17:23:34.432693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432629 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-node-log\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432657 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432685 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.432901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432687 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-node-log\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432711 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-netd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432757 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.432901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432764 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.432901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432803 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432929 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.432995 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-netd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433017 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-cni-dir\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433026 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-dbus\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433058 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433083 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-sys-fs\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433110 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-etc-kubernetes\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.433140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433136 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-conf\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433162 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-sys\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433190 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433216 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-socket-dir-parent\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433241 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-k8s-cni-cncf-io\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433291 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-k8s-cni-cncf-io\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433296 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-etc-kubernetes\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433343 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-sys\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.433488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433442 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-conf\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.433877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433511 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-dbus\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.433877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433855 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.433975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.433947 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-multus-socket-dir-parent\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.434027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434001 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-sys-fs\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.434077 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434033 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-wtmp\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.434077 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434063 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.434175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434087 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-multus\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.434175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434112 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-multus-certs\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.434175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434169 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.434312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434198 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-bin\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.434312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434224 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf40ad1-2c0b-4b84-9242-4108e40de77f-host-slash\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.434312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434252 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-system-cni-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.434312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434269 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-node-exporter-wtmp\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.434312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434278 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmf4g\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434319 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-kubelet\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434351 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-modprobe-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434380 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-root\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434405 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-bin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434433 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-var-lib-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434459 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-etc-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434486 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434511 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-netns\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.434560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434547 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434565 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-multus\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434569 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-host\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434615 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-host\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434655 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-multus-certs\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434617 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-kubelet-config\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434220 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434710 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e-kubelet-config\") pod \"global-pull-secret-syncer-pxq56\" (UID: \"bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e\") " pod="kube-system/global-pull-secret-syncer-pxq56" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434759 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-kubelet\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434869 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-modprobe-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434902 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a23d348-6b2d-4ded-9ff9-543a1e01f6cf-root\") pod \"node-exporter-jmf49\" (UID: \"9a23d348-6b2d-4ded-9ff9-543a1e01f6cf\") " pod="openshift-monitoring/node-exporter-jmf49" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434934 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-var-lib-cni-bin\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434965 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-var-lib-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.434998 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-etc-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435039 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-etc-selinux\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435073 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-run-netns\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435215 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-cni-bin\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435284 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-openvswitch\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.435366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435319 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cnibin\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435600 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-cnibin\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435663 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-socket-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435669 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435717 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf40ad1-2c0b-4b84-9242-4108e40de77f-host-slash\") pod \"iptables-alerter-hq2f5\" (UID: \"9cf40ad1-2c0b-4b84-9242-4108e40de77f\") " pod="openshift-network-operator/iptables-alerter-hq2f5" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435750 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-socket-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435784 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9f458e79-ca1c-4eb2-9ddd-f3daacf6055a-system-cni-dir\") pod \"multus-additional-cni-plugins-dc65z\" (UID: \"9f458e79-ca1c-4eb2-9ddd-f3daacf6055a\") " pod="openshift-multus/multus-additional-cni-plugins-dc65z" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435830 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-registration-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435862 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-device-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435886 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-netns\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435910 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-kubernetes\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435936 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-lib-modules\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435933 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-device-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.435974 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-host-run-netns\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436041 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8a199fd-2c80-44c0-9930-19820009ada9-registration-dir\") pod \"aws-ebs-csi-driver-node-hxv2p\" (UID: \"d8a199fd-2c80-44c0-9930-19820009ada9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hxv2p" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436049 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-lib-modules\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436081 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-slash\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436092 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-kubernetes\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436127 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6627b470-4507-4754-8d06-ef654308f832-crio-socket\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436134 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-slash\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436184 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6627b470-4507-4754-8d06-ef654308f832-crio-socket\") pod \"insights-runtime-extractor-kmkz5\" (UID: \"6627b470-4507-4754-8d06-ef654308f832\") " pod="openshift-insights/insights-runtime-extractor-kmkz5" Nov 02 17:23:34.436220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436195 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436236 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-var-lib-kubelet\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436369 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-var-lib-kubelet\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436405 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436446 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436470 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-hostroot\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436521 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0c38f276-7bda-4a9d-a2b9-3e07385ed119-hostroot\") pod \"multus-bwsqh\" (UID: \"0c38f276-7bda-4a9d-a2b9-3e07385ed119\") " pod="openshift-multus/multus-bwsqh" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436550 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436587 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436610 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436632 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436822 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436885 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.436912 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-ovn\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.437007 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.437469 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysctl-d\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.437553 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.437617 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-ovn\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438140 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438161 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438226 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438292 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysconfig\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-systemd\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438358 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-systemd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438384 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-log-socket\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438416 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b26856a8-af0c-48c4-b6a6-8042e9bea094-host\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438444 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438470 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438560 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438623 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-run-systemd\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438672 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-systemd\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.438754 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e6541049-99e9-453c-9953-8817f491dc4d-etc-sysconfig\") pod \"tuned-dj5gx\" (UID: \"e6541049-99e9-453c-9953-8817f491dc4d\") " pod="openshift-cluster-node-tuning-operator/tuned-dj5gx" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.439211 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b26856a8-af0c-48c4-b6a6-8042e9bea094-host\") pod \"node-ca-w8qzs\" (UID: \"b26856a8-af0c-48c4-b6a6-8042e9bea094\") " pod="openshift-image-registry/node-ca-w8qzs" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.439262 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-log-socket\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.439399 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63987b50-f8b5-42a2-a5cf-50fc6cec635e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-67tdm\" (UID: \"63987b50-f8b5-42a2-a5cf-50fc6cec635e\") " pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.439781 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.440360 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.441395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.440467 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.443391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.441942 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.443391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.442349 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.443391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.442467 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.443391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.442928 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.443391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.443177 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.443391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.443188 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.444252 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.444232 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.454202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.454181 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:23:34.493400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.493370 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmf4g\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g\") pod \"prometheus-k8s-0\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:34.782007 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:34.781974 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:35.264702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:35.264671 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:23:35.268027 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:23:35.267996 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d80a15b_969f_41fc_a78f_ffe9f88a2dea.slice/crio-d8ba4573934809c6ec38b7cc1c02b061b8efeea07f5bcc52d931eaac4ec52b55 WatchSource:0}: Error finding container d8ba4573934809c6ec38b7cc1c02b061b8efeea07f5bcc52d931eaac4ec52b55: Status 404 returned error can't find the container with id d8ba4573934809c6ec38b7cc1c02b061b8efeea07f5bcc52d931eaac4ec52b55 Nov 02 17:23:35.949816 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:23:35.949774 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:23:36.159795 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:36.159756 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9sr25" Nov 02 17:23:36.184264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:36.184048 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" exitCode=0 Nov 02 17:23:36.184264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:36.184090 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} Nov 02 17:23:36.184264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:36.184205 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"d8ba4573934809c6ec38b7cc1c02b061b8efeea07f5bcc52d931eaac4ec52b55"} Nov 02 17:23:36.189040 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:36.189014 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerStarted","Data":"945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88"} Nov 02 17:23:37.548231 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:37.548200 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:37.787883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:37.787847 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:23:38.510352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:38.510314 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:38.510574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:38.510552 5647 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 17:23:38.581536 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:38.581505 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:41.215967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:41.215931 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} Nov 02 17:23:41.216352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:41.216006 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} Nov 02 17:23:42.039280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:42.039243 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:42.045228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:42.045182 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gjj99" Nov 02 17:23:42.199065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:42.199031 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-67tdm" Nov 02 17:23:43.238653 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:43.238620 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} Nov 02 17:23:43.239229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:43.239205 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} Nov 02 17:23:43.433913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:43.433876 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-lwmxf" Nov 02 17:23:44.044779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:44.044727 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:44.247724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:44.247592 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} Nov 02 17:23:44.247724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:44.247686 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerStarted","Data":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} Nov 02 17:23:44.531157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:44.531099 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.870608152 podStartE2EDuration="12.531083018s" totalImagesPullingTime="6.640968852s" totalInitContainerRuntime="19.506014ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:23:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:23:44.479583645 +0000 UTC m=+11.553110863" watchObservedRunningTime="2026-11-02 17:23:44.531083018 +0000 UTC m=+11.604610219" Nov 02 17:23:44.693099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:44.693064 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:23:53.293717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:53.293606 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-wwch5" event={"ID":"4b463fd2-4b01-4441-8d41-dbbe0a440a22","Type":"ContainerStarted","Data":"13bd9ad6c9a1e111026597514f339953759acea58b510b9698b95b8845e4b7d2"} Nov 02 17:23:53.294717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:53.294690 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:53.310460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:53.310427 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-wwch5" Nov 02 17:23:53.907075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:53.907041 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x89xb" Nov 02 17:23:58.587650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:58.587611 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:23:58.592644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:23:58.592618 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f564b5ffb-5scd2" Nov 02 17:24:05.946299 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:05.946261 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:24:06.236253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:06.236162 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:06.289827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:06.289799 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:06.365537 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:06.365503 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:07.791272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:07.791232 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tlgwq" Nov 02 17:24:19.811749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.811700 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:24:19.812332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.812034 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="alertmanager" containerID="cri-o://d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d" gracePeriod=120 Nov 02 17:24:19.812332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.812119 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-metric" containerID="cri-o://81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee" gracePeriod=120 Nov 02 17:24:19.812332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.812180 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="prom-label-proxy" containerID="cri-o://945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88" gracePeriod=120 Nov 02 17:24:19.812332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.812184 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="config-reloader" containerID="cri-o://4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888" gracePeriod=120 Nov 02 17:24:19.812332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.812127 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-web" containerID="cri-o://fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4" gracePeriod=120 Nov 02 17:24:19.812332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:19.812123 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy" containerID="cri-o://087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639" gracePeriod=120 Nov 02 17:24:20.409887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409849 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88" exitCode=0 Nov 02 17:24:20.409887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409882 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee" exitCode=0 Nov 02 17:24:20.409887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409889 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639" exitCode=0 Nov 02 17:24:20.409887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409897 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888" exitCode=0 Nov 02 17:24:20.410170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409902 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d" exitCode=0 Nov 02 17:24:20.410170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409923 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88"} Nov 02 17:24:20.410170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.409986 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee"} Nov 02 17:24:20.410170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.410011 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639"} Nov 02 17:24:20.410170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.410033 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888"} Nov 02 17:24:20.410170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:20.410057 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d"} Nov 02 17:24:21.107099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.107070 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.171862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171772 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.171862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171816 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.171862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171855 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171907 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171933 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171960 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.171986 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172013 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172061 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172105 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172140 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172170 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172210 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") pod \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\" (UID: \"e64a1f95-269d-4654-bef7-e9ba232ce9ee\") " Nov 02 17:24:21.172556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172234 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:24:21.172556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.172473 5647 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-main-db\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.173131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.173048 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:21.173438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.173386 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:21.175276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.175246 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.175639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.175608 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.176204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.176161 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.176319 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.176231 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "kube-api-access-n5cqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:24:21.176575 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.176553 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.176705 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.176667 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:24:21.176860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.176838 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.176937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.176906 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:24:21.180289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.180256 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.188158 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.188129 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config" pod "e64a1f95-269d-4654-bef7-e9ba232ce9ee" (UID: "e64a1f95-269d-4654-bef7-e9ba232ce9ee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:21.273210 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273171 5647 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-cluster-tls-config\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273210 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273202 5647 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-tls-assets\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273230 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-metrics-client-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273240 5647 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-web-config\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273250 5647 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-out\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273260 5647 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e64a1f95-269d-4654-bef7-e9ba232ce9ee-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273273 5647 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273282 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5cqw\" (UniqueName: \"kubernetes.io/projected/e64a1f95-269d-4654-bef7-e9ba232ce9ee-kube-api-access-n5cqw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273292 5647 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-main-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273300 5647 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273311 5647 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.273409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.273319 5647 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e64a1f95-269d-4654-bef7-e9ba232ce9ee-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:21.422687 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.422576 5647 generic.go:356] "Generic (PLEG): container finished" podID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerID="fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4" exitCode=0 Nov 02 17:24:21.422687 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.422653 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4"} Nov 02 17:24:21.422948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.422689 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.422948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.422726 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e64a1f95-269d-4654-bef7-e9ba232ce9ee","Type":"ContainerDied","Data":"94acfe4437d84c18f686cc30b375026f945a6a93da63a79b6ebb88def859a6bb"} Nov 02 17:24:21.422948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.422772 5647 scope.go:122] "RemoveContainer" containerID="945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88" Nov 02 17:24:21.432386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.432265 5647 scope.go:122] "RemoveContainer" containerID="81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee" Nov 02 17:24:21.440070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.440046 5647 scope.go:122] "RemoveContainer" containerID="087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639" Nov 02 17:24:21.447799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.447781 5647 scope.go:122] "RemoveContainer" containerID="fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4" Nov 02 17:24:21.455168 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.455146 5647 scope.go:122] "RemoveContainer" containerID="4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888" Nov 02 17:24:21.462696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.462675 5647 scope.go:122] "RemoveContainer" containerID="d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d" Nov 02 17:24:21.470052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.470032 5647 scope.go:122] "RemoveContainer" containerID="cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3" Nov 02 17:24:21.478116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478085 5647 scope.go:122] "RemoveContainer" containerID="945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88" Nov 02 17:24:21.478419 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.478399 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88\": container with ID starting with 945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88 not found: ID does not exist" containerID="945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88" Nov 02 17:24:21.478463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478430 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88"} err="failed to get container status \"945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88\": rpc error: code = NotFound desc = could not find container \"945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88\": container with ID starting with 945fcf76901aaa87fec97ad87b9dec49c49660bc57abd6a9e417bd5186300a88 not found: ID does not exist" Nov 02 17:24:21.478509 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478463 5647 scope.go:122] "RemoveContainer" containerID="81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee" Nov 02 17:24:21.478691 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.478674 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee\": container with ID starting with 81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee not found: ID does not exist" containerID="81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee" Nov 02 17:24:21.478771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478696 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee"} err="failed to get container status \"81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee\": rpc error: code = NotFound desc = could not find container \"81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee\": container with ID starting with 81a2ee8fd045f86a70f80bbb346dc667e54d272f8e0004bf0ed23b4e890489ee not found: ID does not exist" Nov 02 17:24:21.478771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478710 5647 scope.go:122] "RemoveContainer" containerID="087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639" Nov 02 17:24:21.478934 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.478920 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639\": container with ID starting with 087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639 not found: ID does not exist" containerID="087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639" Nov 02 17:24:21.478967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478939 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639"} err="failed to get container status \"087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639\": rpc error: code = NotFound desc = could not find container \"087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639\": container with ID starting with 087b8cee79f7dabee378ddd02258ae2d047eee5f5c3ad255ae2c1c23a08a1639 not found: ID does not exist" Nov 02 17:24:21.478967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.478950 5647 scope.go:122] "RemoveContainer" containerID="fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4" Nov 02 17:24:21.479227 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.479209 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4\": container with ID starting with fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4 not found: ID does not exist" containerID="fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4" Nov 02 17:24:21.479270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479234 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4"} err="failed to get container status \"fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4\": rpc error: code = NotFound desc = could not find container \"fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4\": container with ID starting with fa5c2cf1fef7667fd49e872ed20ef7882e1f25423fd0b8c106d007c029e2b1a4 not found: ID does not exist" Nov 02 17:24:21.479270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479253 5647 scope.go:122] "RemoveContainer" containerID="4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888" Nov 02 17:24:21.479467 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.479452 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888\": container with ID starting with 4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888 not found: ID does not exist" containerID="4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888" Nov 02 17:24:21.479509 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479480 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888"} err="failed to get container status \"4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888\": rpc error: code = NotFound desc = could not find container \"4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888\": container with ID starting with 4dfd220f8930a04f6f151d8f5973bf75ce5adac7134c0a789675da02a655b888 not found: ID does not exist" Nov 02 17:24:21.479509 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479494 5647 scope.go:122] "RemoveContainer" containerID="d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d" Nov 02 17:24:21.479721 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.479706 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d\": container with ID starting with d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d not found: ID does not exist" containerID="d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d" Nov 02 17:24:21.479869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479724 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d"} err="failed to get container status \"d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d\": rpc error: code = NotFound desc = could not find container \"d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d\": container with ID starting with d8ddf4a3eb52b097bbfe87d136a13df808773fe9a3d255c755787a1c262e366d not found: ID does not exist" Nov 02 17:24:21.479869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479765 5647 scope.go:122] "RemoveContainer" containerID="cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3" Nov 02 17:24:21.479991 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:21.479975 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3\": container with ID starting with cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3 not found: ID does not exist" containerID="cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3" Nov 02 17:24:21.480025 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.479995 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3"} err="failed to get container status \"cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3\": rpc error: code = NotFound desc = could not find container \"cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3\": container with ID starting with cdebc1927b4cb97acd3df52a9b9a5b632eb352e59d127bc22d085b1de72f3ef3 not found: ID does not exist" Nov 02 17:24:21.591706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.591669 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:24:21.598345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.596368 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:24:21.642350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642317 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:24:21.642590 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642579 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="prom-label-proxy" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642592 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="prom-label-proxy" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642610 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="alertmanager" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642614 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="alertmanager" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642622 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-metric" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642628 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-metric" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642635 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642640 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy" Nov 02 17:24:21.642648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642649 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="config-reloader" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642654 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="config-reloader" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642664 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="init-config-reloader" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642669 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="init-config-reloader" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642676 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-web" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642680 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-web" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642746 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="alertmanager" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642753 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-metric" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642760 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642766 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="config-reloader" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642772 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="kube-rbac-proxy-web" Nov 02 17:24:21.642930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.642780 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" containerName="prom-label-proxy" Nov 02 17:24:21.684798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.684703 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.711997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.711964 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:24:21.738348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738310 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 17:24:21.738348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738339 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 17:24:21.738653 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738612 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 17:24:21.738720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738611 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 17:24:21.738720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738652 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 17:24:21.738844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738772 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 17:24:21.738844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738819 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 17:24:21.738946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738900 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 17:24:21.739008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.738983 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hmltn\"" Nov 02 17:24:21.761880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.761842 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 17:24:21.778216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778184 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778222 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778243 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-web-config\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778260 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778388 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778423 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-config-out\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778526 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778446 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778526 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778478 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778579 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778611 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778663 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cll\" (UniqueName: \"kubernetes.io/projected/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-kube-api-access-k8cll\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778684 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.778807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.778715 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879236 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879188 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879236 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879233 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879250 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-web-config\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879275 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879305 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879337 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-config-out\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879370 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879395 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879447 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879470 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879496 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cll\" (UniqueName: \"kubernetes.io/projected/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-kube-api-access-k8cll\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879530 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879574 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.879984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.879963 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.880870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.880846 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.880967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.880907 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.882583 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.882554 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.882693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.882591 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-web-config\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.882693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.882555 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-config-out\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.882875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.882754 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.882973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.882949 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.883516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.883494 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.883853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.883832 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.884131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.884114 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.884358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.884342 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:21.939094 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:21.939030 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cll\" (UniqueName: \"kubernetes.io/projected/5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2-kube-api-access-k8cll\") pod \"alertmanager-main-0\" (UID: \"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:22.087667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:22.087627 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 17:24:22.591929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:22.591907 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 17:24:22.594358 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:24:22.594329 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbd45d5_6de4_4bb2_ba5f_f8e6eefc48c2.slice/crio-e0c231220d8f5cfa22605dc79683282e2f0aa0dad38616cfdc2858de002c586a WatchSource:0}: Error finding container e0c231220d8f5cfa22605dc79683282e2f0aa0dad38616cfdc2858de002c586a: Status 404 returned error can't find the container with id e0c231220d8f5cfa22605dc79683282e2f0aa0dad38616cfdc2858de002c586a Nov 02 17:24:23.078160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.078081 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64a1f95-269d-4654-bef7-e9ba232ce9ee" path="/var/lib/kubelet/pods/e64a1f95-269d-4654-bef7-e9ba232ce9ee/volumes" Nov 02 17:24:23.432679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.432645 5647 generic.go:356] "Generic (PLEG): container finished" podID="5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2" containerID="23ef7ad4aee9b3f0a237073ae8d81077de437ddd73cdbf56507c007a27a58ed4" exitCode=0 Nov 02 17:24:23.432885 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.432726 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerDied","Data":"23ef7ad4aee9b3f0a237073ae8d81077de437ddd73cdbf56507c007a27a58ed4"} Nov 02 17:24:23.432885 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.432838 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"e0c231220d8f5cfa22605dc79683282e2f0aa0dad38616cfdc2858de002c586a"} Nov 02 17:24:23.537208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.537168 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw"] Nov 02 17:24:23.559912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.559886 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.593773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.593747 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-secret-telemeter-client\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.593816 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2r8h\" (UniqueName: \"kubernetes.io/projected/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-kube-api-access-c2r8h\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.593855 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.593903 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.593925 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-metrics-client-ca\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.593985 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-serving-certs-ca-bundle\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.594044 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-federate-client-tls\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.594337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.594160 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-telemeter-client-tls\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.613940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.613902 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 02 17:24:23.614100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.613944 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 02 17:24:23.614100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.613956 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 02 17:24:23.614100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.613984 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 02 17:24:23.614226 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.614133 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fvfxm\"" Nov 02 17:24:23.664936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.664879 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 02 17:24:23.665190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.665170 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 02 17:24:23.672218 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.672189 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw"] Nov 02 17:24:23.695321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695297 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-serving-certs-ca-bundle\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695356 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-federate-client-tls\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695380 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-telemeter-client-tls\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695408 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-secret-telemeter-client\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695423 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2r8h\" (UniqueName: \"kubernetes.io/projected/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-kube-api-access-c2r8h\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695443 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695574 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.695643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.695616 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-metrics-client-ca\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.696074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.696053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-serving-certs-ca-bundle\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.696378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.696356 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-metrics-client-ca\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.696451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.696361 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.698219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.698196 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-telemeter-client-tls\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.698534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.698508 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-secret-telemeter-client\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.698581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.698562 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-federate-client-tls\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.698581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.698574 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.753979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.753951 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2r8h\" (UniqueName: \"kubernetes.io/projected/391e59bd-0f60-41e7-a7db-54f3bf5eddd3-kube-api-access-c2r8h\") pod \"telemeter-client-5f8694f5cb-nt4tw\" (UID: \"391e59bd-0f60-41e7-a7db-54f3bf5eddd3\") " pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:23.939322 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:23.939249 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" Nov 02 17:24:24.439096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.439065 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw"] Nov 02 17:24:24.441001 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.440972 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"a247b93993714e44399c2d613b2202ae7d95e3141b92e3738af9ffb6b2bba105"} Nov 02 17:24:24.441132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.441049 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"2dc9d8e5d34d48c4bb177faf4522a706dd736839131aa3c5f9bc7ffd6c8145c3"} Nov 02 17:24:24.441132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.441098 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"022adb4b696e7802be54914029d5844ded45f2f8411723f2bbfe76c2627cd129"} Nov 02 17:24:24.441253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.441136 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"9563d423cd4030973f8e186c9e41fb4b8ecf78eeb8fe3e2528fa2c740fe4a8bf"} Nov 02 17:24:24.441253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.441174 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"7ea3417f8fefa2dc3a7c3c4662ee6f584207bcc92956df0f5bc954317d87c060"} Nov 02 17:24:24.441253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.441212 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2","Type":"ContainerStarted","Data":"ffaed9b1da301f7f938ecd906cea669aa87c95e5e49ee30cf49d203fe48131c3"} Nov 02 17:24:24.441371 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:24:24.441346 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391e59bd_0f60_41e7_a7db_54f3bf5eddd3.slice/crio-e128eb6a0e5cf46b152c345b805f2ae1f6b42151695f1923905ea7a16f42b0cd WatchSource:0}: Error finding container e128eb6a0e5cf46b152c345b805f2ae1f6b42151695f1923905ea7a16f42b0cd: Status 404 returned error can't find the container with id e128eb6a0e5cf46b152c345b805f2ae1f6b42151695f1923905ea7a16f42b0cd Nov 02 17:24:24.568901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:24.568848 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.553469224 podStartE2EDuration="3.568834083s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.364859ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:24:24.567645088 +0000 UTC m=+51.641172305" watchObservedRunningTime="2026-11-02 17:24:24.568834083 +0000 UTC m=+51.642361286" Nov 02 17:24:25.446174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.446137 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" event={"ID":"391e59bd-0f60-41e7-a7db-54f3bf5eddd3","Type":"ContainerStarted","Data":"e128eb6a0e5cf46b152c345b805f2ae1f6b42151695f1923905ea7a16f42b0cd"} Nov 02 17:24:25.575513 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.575469 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:24:25.576338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.575904 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="prometheus" containerID="cri-o://35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" gracePeriod=600 Nov 02 17:24:25.576338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.576094 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-web" containerID="cri-o://a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" gracePeriod=600 Nov 02 17:24:25.576338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.576143 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" gracePeriod=600 Nov 02 17:24:25.576338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.576093 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy" containerID="cri-o://838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" gracePeriod=600 Nov 02 17:24:25.576338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.576258 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="config-reloader" containerID="cri-o://0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" gracePeriod=600 Nov 02 17:24:25.576338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.576256 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="thanos-sidecar" containerID="cri-o://fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" gracePeriod=600 Nov 02 17:24:25.868903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.868873 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:25.917491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917456 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917509 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917542 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917566 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917592 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917619 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917651 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.917685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917685 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917766 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917795 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917823 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917860 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917896 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917924 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917961 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.917999 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918028 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918061 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918099 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918144 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g\") pod \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\" (UID: \"9d80a15b-969f-41fc-a78f-ffe9f88a2dea\") " Nov 02 17:24:25.918268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918247 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.918968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918376 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.918968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.918435 5647 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:25.919639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.919609 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:24:25.919777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.919617 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.920383 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.920310 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.920727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.920700 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.921344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.921103 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.921344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.921225 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:24:25.921460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.921369 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "kube-api-access-pmf4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:24:25.922273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.921612 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.922587 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.922543 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.923016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.922812 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.923383 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.923243 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:24:25.923776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.923715 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:24:25.924156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.924119 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.924238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.924204 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.924602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.924580 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.924752 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.924715 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.925968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.925942 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:25.936270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:25.936242 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config" pod "9d80a15b-969f-41fc-a78f-ffe9f88a2dea" (UID: "9d80a15b-969f-41fc-a78f-ffe9f88a2dea"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:24:26.019807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019697 5647 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019870 5647 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-metrics-client-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019895 5647 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-metrics-client-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019907 5647 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019921 5647 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019934 5647 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-kube-rbac-proxy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019946 5647 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019955 5647 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-tls-assets\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019964 5647 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019973 5647 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.019984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.019986 5647 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020000 5647 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-config-out\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020017 5647 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-prometheus-k8s-db\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020031 5647 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020045 5647 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020059 5647 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020074 5647 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-secret-grpc-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020087 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmf4g\" (UniqueName: \"kubernetes.io/projected/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-kube-api-access-pmf4g\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.020534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.020101 5647 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d80a15b-969f-41fc-a78f-ffe9f88a2dea-web-config\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:24:26.455235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455195 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" exitCode=0 Nov 02 17:24:26.455235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455228 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" exitCode=0 Nov 02 17:24:26.455235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455240 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" exitCode=0 Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455250 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" exitCode=0 Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455258 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" exitCode=0 Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455267 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" exitCode=0 Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455285 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455323 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455361 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455397 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455430 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455434 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455472 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455513 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} Nov 02 17:24:26.455931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.455555 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d80a15b-969f-41fc-a78f-ffe9f88a2dea","Type":"ContainerDied","Data":"d8ba4573934809c6ec38b7cc1c02b061b8efeea07f5bcc52d931eaac4ec52b55"} Nov 02 17:24:26.467588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.467416 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.477988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.477959 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.486762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.486717 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.495243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.495225 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.503865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.503843 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.513363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.513338 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.522433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.522402 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.522944 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.522915 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.523058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.522960 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} err="failed to get container status \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" Nov 02 17:24:26.523058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.522982 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.523372 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.523332 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.523452 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.523369 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} err="failed to get container status \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" Nov 02 17:24:26.523452 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.523393 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.523764 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.523715 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.523844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.523778 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} err="failed to get container status \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" Nov 02 17:24:26.523844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.523801 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.524120 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.524098 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.524195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.524124 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} err="failed to get container status \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" Nov 02 17:24:26.524195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.524152 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.524459 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.524426 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.524529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.524462 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} err="failed to get container status \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" Nov 02 17:24:26.524529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.524482 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.524858 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.524834 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.524951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.524865 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} err="failed to get container status \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" Nov 02 17:24:26.524951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.524882 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.525190 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:26.525164 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.525263 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.525201 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} err="failed to get container status \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" Nov 02 17:24:26.525263 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.525221 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.525554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.525525 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} err="failed to get container status \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" Nov 02 17:24:26.525554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.525555 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.525867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.525843 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} err="failed to get container status \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" Nov 02 17:24:26.525963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.525869 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.526212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.526186 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} err="failed to get container status \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" Nov 02 17:24:26.526300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.526213 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.526501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.526476 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} err="failed to get container status \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" Nov 02 17:24:26.526501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.526501 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.526941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.526913 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} err="failed to get container status \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" Nov 02 17:24:26.527034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.526942 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.527477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.527449 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} err="failed to get container status \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" Nov 02 17:24:26.527477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.527477 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.528118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.528091 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} err="failed to get container status \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" Nov 02 17:24:26.528118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.528118 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.528443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.528407 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} err="failed to get container status \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" Nov 02 17:24:26.528443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.528433 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.528829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.528708 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} err="failed to get container status \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" Nov 02 17:24:26.528829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.528749 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.529073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529048 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} err="failed to get container status \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" Nov 02 17:24:26.529073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529072 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.529299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529271 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} err="failed to get container status \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" Nov 02 17:24:26.529379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529300 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.529523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529505 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} err="failed to get container status \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" Nov 02 17:24:26.529593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529524 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.529716 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529698 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} err="failed to get container status \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" Nov 02 17:24:26.529807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529717 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.529951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529932 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} err="failed to get container status \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" Nov 02 17:24:26.530030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.529951 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.530152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530130 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} err="failed to get container status \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" Nov 02 17:24:26.530207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530168 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.530381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530362 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} err="failed to get container status \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" Nov 02 17:24:26.530435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530400 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.530617 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530597 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} err="failed to get container status \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" Nov 02 17:24:26.530667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530618 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.531026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.530999 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} err="failed to get container status \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" Nov 02 17:24:26.531026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531024 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.531333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531309 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} err="failed to get container status \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" Nov 02 17:24:26.531404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531335 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.531648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531617 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} err="failed to get container status \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" Nov 02 17:24:26.531648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531647 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.531961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531938 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} err="failed to get container status \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" Nov 02 17:24:26.532048 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.531961 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.532297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.532270 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} err="failed to get container status \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" Nov 02 17:24:26.532297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.532294 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.532565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.532526 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} err="failed to get container status \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" Nov 02 17:24:26.532565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.532548 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.532810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.532791 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} err="failed to get container status \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" Nov 02 17:24:26.532810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.532810 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.533120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.533098 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} err="failed to get container status \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" Nov 02 17:24:26.533209 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.533120 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.533428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.533402 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} err="failed to get container status \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" Nov 02 17:24:26.533428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.533428 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.533719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.533695 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} err="failed to get container status \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" Nov 02 17:24:26.533822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.533720 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.534082 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.534054 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} err="failed to get container status \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" Nov 02 17:24:26.534159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.534083 5647 scope.go:122] "RemoveContainer" containerID="4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6" Nov 02 17:24:26.534329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.534302 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6"} err="failed to get container status \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": rpc error: code = NotFound desc = could not find container \"4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6\": container with ID starting with 4aa623cedb7f5ff63287f711d7da7d428abc53fa171088c2973610faddb984a6 not found: ID does not exist" Nov 02 17:24:26.534329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.534326 5647 scope.go:122] "RemoveContainer" containerID="838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59" Nov 02 17:24:26.534628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.534602 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59"} err="failed to get container status \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": rpc error: code = NotFound desc = could not find container \"838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59\": container with ID starting with 838d9c0612f507f361649c26e0217e074ea412fdce789a99ec05b3531fd4fb59 not found: ID does not exist" Nov 02 17:24:26.534722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.534629 5647 scope.go:122] "RemoveContainer" containerID="a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5" Nov 02 17:24:26.538018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.537989 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5"} err="failed to get container status \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": rpc error: code = NotFound desc = could not find container \"a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5\": container with ID starting with a880aabdd5728bd8ac05be91525279cb0ddbf0a522da77fcab78200ea6b09ad5 not found: ID does not exist" Nov 02 17:24:26.538018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538015 5647 scope.go:122] "RemoveContainer" containerID="fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d" Nov 02 17:24:26.538365 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538337 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d"} err="failed to get container status \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": rpc error: code = NotFound desc = could not find container \"fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d\": container with ID starting with fc622ac944cd2f94226d34afa397034d5a1f2aa36d276147d98ab9b6f6fb187d not found: ID does not exist" Nov 02 17:24:26.538365 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538364 5647 scope.go:122] "RemoveContainer" containerID="0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2" Nov 02 17:24:26.538691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538667 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2"} err="failed to get container status \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": rpc error: code = NotFound desc = could not find container \"0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2\": container with ID starting with 0c2a9e4354c79cee0ea68dc6eb26e5a4b671f90bc92f5cfe900729aa973f46a2 not found: ID does not exist" Nov 02 17:24:26.538790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538691 5647 scope.go:122] "RemoveContainer" containerID="35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb" Nov 02 17:24:26.538990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538967 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb"} err="failed to get container status \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": rpc error: code = NotFound desc = could not find container \"35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb\": container with ID starting with 35f4b7e5687af053e734245de27ddb2e9ec9f00a4f150e929b622484c6515acb not found: ID does not exist" Nov 02 17:24:26.539055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.538990 5647 scope.go:122] "RemoveContainer" containerID="a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d" Nov 02 17:24:26.539291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.539268 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d"} err="failed to get container status \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": rpc error: code = NotFound desc = could not find container \"a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d\": container with ID starting with a31508fe0c8cccd5c68fcef17a6b615c6f0a5bfa628c5e178d25c27ab2345c5d not found: ID does not exist" Nov 02 17:24:26.626704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.626518 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:24:26.677198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.677166 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:24:26.677683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.677656 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:24:26.678079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678060 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-thanos" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678082 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-thanos" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678108 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="init-config-reloader" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678118 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="init-config-reloader" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678131 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="prometheus" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678139 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="prometheus" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678159 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678168 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy" Nov 02 17:24:26.678181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678181 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="config-reloader" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678189 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="config-reloader" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678209 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="thanos-sidecar" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678217 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="thanos-sidecar" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678228 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-web" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678233 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-web" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678294 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="prometheus" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678303 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678309 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="config-reloader" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678322 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="thanos-sidecar" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678333 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-web" Nov 02 17:24:26.678506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.678342 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" containerName="kube-rbac-proxy-thanos" Nov 02 17:24:26.705689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.703645 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725027 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725244 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725285 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725311 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725341 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725377 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725401 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725460 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725489 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725516 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725544 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725568 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725591 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725615 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfr54\" (UniqueName: \"kubernetes.io/projected/2d999266-4195-4a21-a31e-e832f991e728-kube-api-access-cfr54\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725643 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725664 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d999266-4195-4a21-a31e-e832f991e728-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725678 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-config\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725698 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d999266-4195-4a21-a31e-e832f991e728-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725714 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.725847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.725756 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.757841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.757809 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 02 17:24:26.758356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.758336 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 02 17:24:26.758692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.758673 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 02 17:24:26.758944 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.758928 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 02 17:24:26.808785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.808752 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 02 17:24:26.808785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.808759 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 02 17:24:26.808785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.808753 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 02 17:24:26.809008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.808797 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 02 17:24:26.809008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.808813 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 02 17:24:26.809696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809672 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 02 17:24:26.809854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809707 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 02 17:24:26.809854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809707 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fvvhud3u970td\"" Nov 02 17:24:26.809854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809748 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 02 17:24:26.809854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809762 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 02 17:24:26.809854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809782 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 02 17:24:26.809854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.809823 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sc9pn\"" Nov 02 17:24:26.810678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.810662 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:24:26.826981 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.826948 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.826981 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.826979 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827002 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827023 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827219 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827297 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827339 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827414 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827368 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827414 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827389 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827414 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827441 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827473 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfr54\" (UniqueName: \"kubernetes.io/projected/2d999266-4195-4a21-a31e-e832f991e728-kube-api-access-cfr54\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827495 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827525 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d999266-4195-4a21-a31e-e832f991e728-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827555 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-config\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827584 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d999266-4195-4a21-a31e-e832f991e728-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827609 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827639 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827684 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.827824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827708 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.828090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827886 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.828090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.827955 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.829046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.828573 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.829171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.829122 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.830783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.830759 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.831330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.831303 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.831480 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.831438 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.831544 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.831492 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d999266-4195-4a21-a31e-e832f991e728-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.831697 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.831670 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.832420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.832251 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.832420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.832325 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2d999266-4195-4a21-a31e-e832f991e728-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.832420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.832379 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d999266-4195-4a21-a31e-e832f991e728-config-out\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.833173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.833105 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.834033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.834010 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.834388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.834357 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-config\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.834658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.834636 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.834768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.834644 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.834873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.834856 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-web-config\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.835576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.835560 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2d999266-4195-4a21-a31e-e832f991e728-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:26.888698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:26.888672 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfr54\" (UniqueName: \"kubernetes.io/projected/2d999266-4195-4a21-a31e-e832f991e728-kube-api-access-cfr54\") pod \"prometheus-k8s-0\" (UID: \"2d999266-4195-4a21-a31e-e832f991e728\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:27.078656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:27.078584 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d80a15b-969f-41fc-a78f-ffe9f88a2dea" path="/var/lib/kubelet/pods/9d80a15b-969f-41fc-a78f-ffe9f88a2dea/volumes" Nov 02 17:24:27.172128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:27.172103 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:27.460666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:27.460633 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" event={"ID":"391e59bd-0f60-41e7-a7db-54f3bf5eddd3","Type":"ContainerStarted","Data":"7d8489b6d53a609f25fb8f8140d8b95df1f58db7ffbd3e58833ad166f74cd10a"} Nov 02 17:24:27.461122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:27.460697 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" event={"ID":"391e59bd-0f60-41e7-a7db-54f3bf5eddd3","Type":"ContainerStarted","Data":"8fba65979f0d7f7976855069fb0ebff645dfa6f24c7e4700e8561b928a049995"} Nov 02 17:24:27.684401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:27.684263 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 02 17:24:27.687046 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:24:27.687014 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d999266_4195_4a21_a31e_e832f991e728.slice/crio-8e271a9f1e0b76e2572b55782acb4b889c2781a8f72f10057cc3bb98d0b77543 WatchSource:0}: Error finding container 8e271a9f1e0b76e2572b55782acb4b889c2781a8f72f10057cc3bb98d0b77543: Status 404 returned error can't find the container with id 8e271a9f1e0b76e2572b55782acb4b889c2781a8f72f10057cc3bb98d0b77543 Nov 02 17:24:28.471390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:28.471351 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" event={"ID":"391e59bd-0f60-41e7-a7db-54f3bf5eddd3","Type":"ContainerStarted","Data":"c5246a8ee94ef5b202275a2aed1baa18cd0f069aa7d54a2acde8aa7a3cef4ef7"} Nov 02 17:24:28.472935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:28.472914 5647 generic.go:356] "Generic (PLEG): container finished" podID="2d999266-4195-4a21-a31e-e832f991e728" containerID="087e0b060073a6e6a6a45114db8ce80536c5bc401e4726318c9712f1d8614804" exitCode=0 Nov 02 17:24:28.473016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:28.472952 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerDied","Data":"087e0b060073a6e6a6a45114db8ce80536c5bc401e4726318c9712f1d8614804"} Nov 02 17:24:28.473016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:28.472994 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"8e271a9f1e0b76e2572b55782acb4b889c2781a8f72f10057cc3bb98d0b77543"} Nov 02 17:24:28.594155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:28.593720 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5f8694f5cb-nt4tw" podStartSLOduration=2.903944445 podStartE2EDuration="5.593701148s" totalImagesPullingTime="2.689756703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:24:28.59289319 +0000 UTC m=+55.666420416" watchObservedRunningTime="2026-11-02 17:24:28.593701148 +0000 UTC m=+55.667228346" Nov 02 17:24:29.480385 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.480346 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"f8a5bb027f173bd2ff270f982553bf00d0988ea2186a46eb5332d4242d660631"} Nov 02 17:24:29.480907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.480425 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"abbc4b3a03533d93bf3e0153bf85e7b0e2bd7f59c4e27bb7bb03ca37c79e4461"} Nov 02 17:24:29.480907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.480454 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"a969e59a7358e96578c32a7e2379c0456b30b4424fea05a8c56c2da3c57c5053"} Nov 02 17:24:29.480907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.480477 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"12c6deb74879027c36f822f2f8cf9509c0834ced232f1ef156adeec1fb46eb77"} Nov 02 17:24:29.480907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.480506 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"f2a972247b20f3592af6c5be218470ecbfb1eff903b5dc742994ffeaabc7810d"} Nov 02 17:24:29.480907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.480538 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2d999266-4195-4a21-a31e-e832f991e728","Type":"ContainerStarted","Data":"8fbfa801c38b0b8d18e43142efe41454fc043bed4401ea59ed08884a254f034d"} Nov 02 17:24:29.609531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:29.609480 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.583417307 podStartE2EDuration="3.609465282s" totalImagesPullingTime="0s" totalInitContainerRuntime="26.047975ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:24:29.608451419 +0000 UTC m=+56.681978646" watchObservedRunningTime="2026-11-02 17:24:29.609465282 +0000 UTC m=+56.682992476" Nov 02 17:24:32.172382 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:32.172345 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:35.946494 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:24:35.946451 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:24:46.652419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:46.652376 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:46.668640 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:46.668616 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:24:47.189066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:24:47.189038 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 02 17:25:05.945996 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:25:05.945955 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:25:35.946773 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:25:35.946673 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:26:05.945812 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:26:05.945775 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:26:35.945671 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:26:35.945629 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:27:05.946207 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:27:05.946164 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:27:35.945908 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:27:35.945871 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:27:39.784403 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.784368 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm"] Nov 02 17:27:39.787697 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.787681 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.848621 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.848576 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.848826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.848632 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.848826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.848655 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.848826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.848755 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkvr\" (UniqueName: \"kubernetes.io/projected/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-kube-api-access-xbkvr\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.848826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.848796 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.848984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.848878 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.918512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.918480 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 17:27:39.918690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.918534 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 17:27:39.918690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.918484 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 17:27:39.918690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.918603 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 17:27:39.918690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.918607 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-z8nm9\"" Nov 02 17:27:39.950065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950036 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.950232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950079 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.950232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950205 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.950318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950281 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.950318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950308 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.950409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950331 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkvr\" (UniqueName: \"kubernetes.io/projected/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-kube-api-access-xbkvr\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.950586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.950559 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-manifests\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.952611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.952590 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-tmp\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.952884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.952863 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-metrics-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.952968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.952910 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.952968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.952941 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:39.969320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:39.969292 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 17:27:40.014180 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:40.014144 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkvr\" (UniqueName: \"kubernetes.io/projected/43426ca7-9e9a-4e6b-a6f0-f83a014146b2-kube-api-access-xbkvr\") pod \"opendatahub-operator-controller-manager-89b98757-zwpfm\" (UID: \"43426ca7-9e9a-4e6b-a6f0-f83a014146b2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:40.044585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:40.044512 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm"] Nov 02 17:27:40.102911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:40.102870 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:40.601332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:40.601307 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm"] Nov 02 17:27:40.604138 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:27:40.604109 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43426ca7_9e9a_4e6b_a6f0_f83a014146b2.slice/crio-a9b430fa8c50f0e427c12319e5a6d4421f204ed3a073db2993fa654bc1c19b83 WatchSource:0}: Error finding container a9b430fa8c50f0e427c12319e5a6d4421f204ed3a073db2993fa654bc1c19b83: Status 404 returned error can't find the container with id a9b430fa8c50f0e427c12319e5a6d4421f204ed3a073db2993fa654bc1c19b83 Nov 02 17:27:41.213980 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:41.213942 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" event={"ID":"43426ca7-9e9a-4e6b-a6f0-f83a014146b2","Type":"ContainerStarted","Data":"a9b430fa8c50f0e427c12319e5a6d4421f204ed3a073db2993fa654bc1c19b83"} Nov 02 17:27:44.227354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.227322 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" event={"ID":"43426ca7-9e9a-4e6b-a6f0-f83a014146b2","Type":"ContainerStarted","Data":"430acb4ae62c954c7e491dc4983fdf51b3c1387306015531fbe2cc576ecfa04d"} Nov 02 17:27:44.753943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.753911 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl"] Nov 02 17:27:44.757177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.757160 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:44.816494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.816463 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 17:27:44.816674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.816543 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 17:27:44.816836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.816808 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-dzhfg\"" Nov 02 17:27:44.817365 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.817341 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl"] Nov 02 17:27:44.896470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.896381 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcrfh\" (UniqueName: \"kubernetes.io/projected/06b23ba2-310c-4f65-a7ed-5c616b56dc47-kube-api-access-fcrfh\") pod \"jobset-operator-674cd59dc5-sltjl\" (UID: \"06b23ba2-310c-4f65-a7ed-5c616b56dc47\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:44.896640 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.896572 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06b23ba2-310c-4f65-a7ed-5c616b56dc47-tmp\") pod \"jobset-operator-674cd59dc5-sltjl\" (UID: \"06b23ba2-310c-4f65-a7ed-5c616b56dc47\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:44.997914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.997872 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06b23ba2-310c-4f65-a7ed-5c616b56dc47-tmp\") pod \"jobset-operator-674cd59dc5-sltjl\" (UID: \"06b23ba2-310c-4f65-a7ed-5c616b56dc47\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:44.998090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.997920 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcrfh\" (UniqueName: \"kubernetes.io/projected/06b23ba2-310c-4f65-a7ed-5c616b56dc47-kube-api-access-fcrfh\") pod \"jobset-operator-674cd59dc5-sltjl\" (UID: \"06b23ba2-310c-4f65-a7ed-5c616b56dc47\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:44.998285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:44.998265 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06b23ba2-310c-4f65-a7ed-5c616b56dc47-tmp\") pod \"jobset-operator-674cd59dc5-sltjl\" (UID: \"06b23ba2-310c-4f65-a7ed-5c616b56dc47\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:45.057311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:45.057266 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcrfh\" (UniqueName: \"kubernetes.io/projected/06b23ba2-310c-4f65-a7ed-5c616b56dc47-kube-api-access-fcrfh\") pod \"jobset-operator-674cd59dc5-sltjl\" (UID: \"06b23ba2-310c-4f65-a7ed-5c616b56dc47\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:45.082237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:45.082211 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" Nov 02 17:27:45.233138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:45.233060 5647 generic.go:356] "Generic (PLEG): container finished" podID="43426ca7-9e9a-4e6b-a6f0-f83a014146b2" containerID="430acb4ae62c954c7e491dc4983fdf51b3c1387306015531fbe2cc576ecfa04d" exitCode=0 Nov 02 17:27:45.233532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:45.233144 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" event={"ID":"43426ca7-9e9a-4e6b-a6f0-f83a014146b2","Type":"ContainerDied","Data":"430acb4ae62c954c7e491dc4983fdf51b3c1387306015531fbe2cc576ecfa04d"} Nov 02 17:27:45.537527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:45.537506 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl"] Nov 02 17:27:45.540292 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:27:45.540268 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06b23ba2_310c_4f65_a7ed_5c616b56dc47.slice/crio-15cf3520600acda553b317373f9dfae4fc65c2108e59a95428c5663f72def590 WatchSource:0}: Error finding container 15cf3520600acda553b317373f9dfae4fc65c2108e59a95428c5663f72def590: Status 404 returned error can't find the container with id 15cf3520600acda553b317373f9dfae4fc65c2108e59a95428c5663f72def590 Nov 02 17:27:46.239386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:46.239344 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" event={"ID":"43426ca7-9e9a-4e6b-a6f0-f83a014146b2","Type":"ContainerStarted","Data":"192ac5a2c7c055b94fc6a2d9b6b5d7297410fafee34d8744b0952056f99ab378"} Nov 02 17:27:46.239873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:46.239542 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:27:46.240761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:46.240721 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" event={"ID":"06b23ba2-310c-4f65-a7ed-5c616b56dc47","Type":"ContainerStarted","Data":"15cf3520600acda553b317373f9dfae4fc65c2108e59a95428c5663f72def590"} Nov 02 17:27:46.381285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:46.381231 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" podStartSLOduration=2.660548651 podStartE2EDuration="7.381217962s" totalImagesPullingTime="4.280252716s" totalInitContainerRuntime="440.416595ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:46.378692655 +0000 UTC m=+253.452219873" watchObservedRunningTime="2026-11-02 17:27:46.381217962 +0000 UTC m=+253.454745159" Nov 02 17:27:50.259114 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:50.259078 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" event={"ID":"06b23ba2-310c-4f65-a7ed-5c616b56dc47","Type":"ContainerStarted","Data":"4e14c153e5f20e6f856c156c78ae05c02f22bde90959c08108937123d7a968d2"} Nov 02 17:27:50.377607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:50.377551 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-sltjl" podStartSLOduration=2.510031258 podStartE2EDuration="6.377533592s" totalImagesPullingTime="3.867502334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:27:50.376925432 +0000 UTC m=+257.450452636" watchObservedRunningTime="2026-11-02 17:27:50.377533592 +0000 UTC m=+257.451060794" Nov 02 17:27:57.247484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:27:57.247446 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-89b98757-zwpfm" Nov 02 17:28:05.024539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.024504 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj"] Nov 02 17:28:05.031031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.031009 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.085052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.085019 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 02 17:28:05.085204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.085075 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 02 17:28:05.087927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.087895 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxgff\" (UniqueName: \"kubernetes.io/projected/2243eaa8-a7d5-4409-a8c4-404ff8690ce7-kube-api-access-vxgff\") pod \"cert-manager-cainjector-5c57bbd4fc-24pkj\" (UID: \"2243eaa8-a7d5-4409-a8c4-404ff8690ce7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.088162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.088131 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2243eaa8-a7d5-4409-a8c4-404ff8690ce7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-24pkj\" (UID: \"2243eaa8-a7d5-4409-a8c4-404ff8690ce7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.136071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.136044 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cc9n5\"" Nov 02 17:28:05.137141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.137118 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj"] Nov 02 17:28:05.189505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.189476 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxgff\" (UniqueName: \"kubernetes.io/projected/2243eaa8-a7d5-4409-a8c4-404ff8690ce7-kube-api-access-vxgff\") pod \"cert-manager-cainjector-5c57bbd4fc-24pkj\" (UID: \"2243eaa8-a7d5-4409-a8c4-404ff8690ce7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.189647 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.189523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2243eaa8-a7d5-4409-a8c4-404ff8690ce7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-24pkj\" (UID: \"2243eaa8-a7d5-4409-a8c4-404ff8690ce7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.249436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.249402 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2243eaa8-a7d5-4409-a8c4-404ff8690ce7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-24pkj\" (UID: \"2243eaa8-a7d5-4409-a8c4-404ff8690ce7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.249583 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.249409 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxgff\" (UniqueName: \"kubernetes.io/projected/2243eaa8-a7d5-4409-a8c4-404ff8690ce7-kube-api-access-vxgff\") pod \"cert-manager-cainjector-5c57bbd4fc-24pkj\" (UID: \"2243eaa8-a7d5-4409-a8c4-404ff8690ce7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.342534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.342502 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" Nov 02 17:28:05.715177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:05.715104 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj"] Nov 02 17:28:05.718010 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:05.717977 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2243eaa8_a7d5_4409_a8c4_404ff8690ce7.slice/crio-5c9c374db4fa872fdc477af418457c33a5ab7db2db3c74cfb9bb8bd920af5544 WatchSource:0}: Error finding container 5c9c374db4fa872fdc477af418457c33a5ab7db2db3c74cfb9bb8bd920af5544: Status 404 returned error can't find the container with id 5c9c374db4fa872fdc477af418457c33a5ab7db2db3c74cfb9bb8bd920af5544 Nov 02 17:28:05.945838 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:28:05.945797 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:28:06.323461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:06.323425 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" event={"ID":"2243eaa8-a7d5-4409-a8c4-404ff8690ce7","Type":"ContainerStarted","Data":"5c9c374db4fa872fdc477af418457c33a5ab7db2db3c74cfb9bb8bd920af5544"} Nov 02 17:28:10.343112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:10.343075 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" event={"ID":"2243eaa8-a7d5-4409-a8c4-404ff8690ce7","Type":"ContainerStarted","Data":"6e39943b1f3202aa8d03d7eeee4a49425ae7183860e47c6deb2a2f94934dcb01"} Nov 02 17:28:10.470682 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:10.470562 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-24pkj" podStartSLOduration=2.080520186 podStartE2EDuration="6.470546881s" totalImagesPullingTime="4.390026695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:10.469146688 +0000 UTC m=+277.542673891" watchObservedRunningTime="2026-11-02 17:28:10.470546881 +0000 UTC m=+277.544074084" Nov 02 17:28:18.301862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.301781 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-9wlhm"] Nov 02 17:28:18.307584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.307558 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.364161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.364121 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9wlhm"] Nov 02 17:28:18.413606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.413573 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-s22jb\"" Nov 02 17:28:18.413834 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.413811 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/435f0b3d-3a53-4c85-b1bd-d328e61d3c37-bound-sa-token\") pod \"cert-manager-597b95cb84-9wlhm\" (UID: \"435f0b3d-3a53-4c85-b1bd-d328e61d3c37\") " pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.413891 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.413858 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6vc\" (UniqueName: \"kubernetes.io/projected/435f0b3d-3a53-4c85-b1bd-d328e61d3c37-kube-api-access-dt6vc\") pod \"cert-manager-597b95cb84-9wlhm\" (UID: \"435f0b3d-3a53-4c85-b1bd-d328e61d3c37\") " pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.515170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.515144 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/435f0b3d-3a53-4c85-b1bd-d328e61d3c37-bound-sa-token\") pod \"cert-manager-597b95cb84-9wlhm\" (UID: \"435f0b3d-3a53-4c85-b1bd-d328e61d3c37\") " pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.515324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.515187 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6vc\" (UniqueName: \"kubernetes.io/projected/435f0b3d-3a53-4c85-b1bd-d328e61d3c37-kube-api-access-dt6vc\") pod \"cert-manager-597b95cb84-9wlhm\" (UID: \"435f0b3d-3a53-4c85-b1bd-d328e61d3c37\") " pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.577878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.577759 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/435f0b3d-3a53-4c85-b1bd-d328e61d3c37-bound-sa-token\") pod \"cert-manager-597b95cb84-9wlhm\" (UID: \"435f0b3d-3a53-4c85-b1bd-d328e61d3c37\") " pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.578771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.578751 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6vc\" (UniqueName: \"kubernetes.io/projected/435f0b3d-3a53-4c85-b1bd-d328e61d3c37-kube-api-access-dt6vc\") pod \"cert-manager-597b95cb84-9wlhm\" (UID: \"435f0b3d-3a53-4c85-b1bd-d328e61d3c37\") " pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.618651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.618624 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-9wlhm" Nov 02 17:28:18.989894 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:18.989860 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-9wlhm"] Nov 02 17:28:18.992777 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:18.992717 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod435f0b3d_3a53_4c85_b1bd_d328e61d3c37.slice/crio-a3bcde10c12ef4a9cb29c84896bc1b0cd6c1b86ea0e498d3398590d44620e010 WatchSource:0}: Error finding container a3bcde10c12ef4a9cb29c84896bc1b0cd6c1b86ea0e498d3398590d44620e010: Status 404 returned error can't find the container with id a3bcde10c12ef4a9cb29c84896bc1b0cd6c1b86ea0e498d3398590d44620e010 Nov 02 17:28:19.381166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:19.381132 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9wlhm" event={"ID":"435f0b3d-3a53-4c85-b1bd-d328e61d3c37","Type":"ContainerStarted","Data":"15b881598f649fe0849208fb0f4bfdf1f5292bb74a12553732c4d3d0611d6264"} Nov 02 17:28:19.381166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:19.381172 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-9wlhm" event={"ID":"435f0b3d-3a53-4c85-b1bd-d328e61d3c37","Type":"ContainerStarted","Data":"a3bcde10c12ef4a9cb29c84896bc1b0cd6c1b86ea0e498d3398590d44620e010"} Nov 02 17:28:19.495722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:19.495669 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-9wlhm" podStartSLOduration=1.495655052 podStartE2EDuration="1.495655052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:19.495262474 +0000 UTC m=+286.568789674" watchObservedRunningTime="2026-11-02 17:28:19.495655052 +0000 UTC m=+286.569182252" Nov 02 17:28:20.857847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.857811 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq"] Nov 02 17:28:20.860308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.860291 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:20.937693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.937659 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-tls-certs\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:20.937917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.937873 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-tmp\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:20.937917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.937910 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49g2c\" (UniqueName: \"kubernetes.io/projected/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-kube-api-access-49g2c\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:20.964170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.964136 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 17:28:20.964327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.964177 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 17:28:20.964327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.964204 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g78m2\"" Nov 02 17:28:20.964620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.964598 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 17:28:20.964775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.964685 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 17:28:20.965157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:20.965136 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq"] Nov 02 17:28:21.039402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.039373 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-tmp\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.039579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.039406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49g2c\" (UniqueName: \"kubernetes.io/projected/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-kube-api-access-49g2c\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.039579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.039543 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-tls-certs\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.041793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.041767 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-tmp\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.042083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.042065 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-tls-certs\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.108914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.108840 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49g2c\" (UniqueName: \"kubernetes.io/projected/1e314e9c-09e2-41fd-a44c-0cc6e5213d11-kube-api-access-49g2c\") pod \"kube-auth-proxy-5d87ddd77c-wgvlq\" (UID: \"1e314e9c-09e2-41fd-a44c-0cc6e5213d11\") " pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.171388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.171362 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" Nov 02 17:28:21.574849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:21.574711 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq"] Nov 02 17:28:21.577446 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:21.577416 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e314e9c_09e2_41fd_a44c_0cc6e5213d11.slice/crio-66657c140c5dedb61ba96b949f86bb259061b6a8459ac808345dbae0e148d11f WatchSource:0}: Error finding container 66657c140c5dedb61ba96b949f86bb259061b6a8459ac808345dbae0e148d11f: Status 404 returned error can't find the container with id 66657c140c5dedb61ba96b949f86bb259061b6a8459ac808345dbae0e148d11f Nov 02 17:28:22.394089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:22.394052 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" event={"ID":"1e314e9c-09e2-41fd-a44c-0cc6e5213d11","Type":"ContainerStarted","Data":"66657c140c5dedb61ba96b949f86bb259061b6a8459ac808345dbae0e148d11f"} Nov 02 17:28:23.464476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.464437 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:23.468312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.468288 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.524846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.524810 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 17:28:23.524846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.524840 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 17:28:23.525093 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.524955 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 17:28:23.566576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.566540 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9l7\" (UniqueName: \"kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7\") pod \"limitador-limitador-74dfcfb894-9rhdw\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.566751 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.566695 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file\") pod \"limitador-limitador-74dfcfb894-9rhdw\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.575699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.575646 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cvtld\"" Nov 02 17:28:23.576815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.576788 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:23.668239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.668206 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9l7\" (UniqueName: \"kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7\") pod \"limitador-limitador-74dfcfb894-9rhdw\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.668420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.668293 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file\") pod \"limitador-limitador-74dfcfb894-9rhdw\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.669002 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.668966 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file\") pod \"limitador-limitador-74dfcfb894-9rhdw\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.714355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.714322 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:23.729549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.729391 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9l7\" (UniqueName: \"kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7\") pod \"limitador-limitador-74dfcfb894-9rhdw\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:23.782995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:23.782959 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:24.184632 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:24.184584 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:24.188072 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:24.188039 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b2d4f9_4b40_411f_b365_51d2e171a85a.slice/crio-2223c21bbe226662114379d7bfde765a26a00f6facd9e380cafae725655f645b WatchSource:0}: Error finding container 2223c21bbe226662114379d7bfde765a26a00f6facd9e380cafae725655f645b: Status 404 returned error can't find the container with id 2223c21bbe226662114379d7bfde765a26a00f6facd9e380cafae725655f645b Nov 02 17:28:24.405550 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:24.405503 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" event={"ID":"e6b2d4f9-4b40-411f-b365-51d2e171a85a","Type":"ContainerStarted","Data":"2223c21bbe226662114379d7bfde765a26a00f6facd9e380cafae725655f645b"} Nov 02 17:28:24.965577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:24.965540 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qnpf"] Nov 02 17:28:24.968890 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:24.968874 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:28:24.969014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:24.968994 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:24.971845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:24.971825 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.022768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.022715 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jm2sm\"" Nov 02 17:28:25.024162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.024139 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qnpf"] Nov 02 17:28:25.082328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.082294 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/069d21b8-f43f-465f-870f-1da8bfc01136-config-file\") pod \"limitador-limitador-7c9b87d574-dzrt7\" (UID: \"069d21b8-f43f-465f-870f-1da8bfc01136\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.082525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.082422 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf7s6\" (UniqueName: \"kubernetes.io/projected/069d21b8-f43f-465f-870f-1da8bfc01136-kube-api-access-sf7s6\") pod \"limitador-limitador-7c9b87d574-dzrt7\" (UID: \"069d21b8-f43f-465f-870f-1da8bfc01136\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.082525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.082493 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92zcr\" (UniqueName: \"kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr\") pod \"authorino-57f7f8659f-4qnpf\" (UID: \"ba9cfaee-932f-42bc-9e42-cc23546474e0\") " pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:25.086565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.086541 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:28:25.183999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.183973 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/069d21b8-f43f-465f-870f-1da8bfc01136-config-file\") pod \"limitador-limitador-7c9b87d574-dzrt7\" (UID: \"069d21b8-f43f-465f-870f-1da8bfc01136\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.184174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.184020 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf7s6\" (UniqueName: \"kubernetes.io/projected/069d21b8-f43f-465f-870f-1da8bfc01136-kube-api-access-sf7s6\") pod \"limitador-limitador-7c9b87d574-dzrt7\" (UID: \"069d21b8-f43f-465f-870f-1da8bfc01136\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.184174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.184053 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92zcr\" (UniqueName: \"kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr\") pod \"authorino-57f7f8659f-4qnpf\" (UID: \"ba9cfaee-932f-42bc-9e42-cc23546474e0\") " pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:25.184586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.184568 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/069d21b8-f43f-465f-870f-1da8bfc01136-config-file\") pod \"limitador-limitador-7c9b87d574-dzrt7\" (UID: \"069d21b8-f43f-465f-870f-1da8bfc01136\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.243144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.243065 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92zcr\" (UniqueName: \"kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr\") pod \"authorino-57f7f8659f-4qnpf\" (UID: \"ba9cfaee-932f-42bc-9e42-cc23546474e0\") " pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:25.243281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.243147 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf7s6\" (UniqueName: \"kubernetes.io/projected/069d21b8-f43f-465f-870f-1da8bfc01136-kube-api-access-sf7s6\") pod \"limitador-limitador-7c9b87d574-dzrt7\" (UID: \"069d21b8-f43f-465f-870f-1da8bfc01136\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.278012 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.277978 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:28:25.282115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.282099 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:25.288015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.287998 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:25.411529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.411491 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" event={"ID":"1e314e9c-09e2-41fd-a44c-0cc6e5213d11","Type":"ContainerStarted","Data":"b6f11ee2fbf9798e53993d3a04bd081cf2686b276358f5161aef72a474fb4e90"} Nov 02 17:28:25.527176 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.527079 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d87ddd77c-wgvlq" podStartSLOduration=2.589634302 podStartE2EDuration="5.527063426s" totalImagesPullingTime="2.937429124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:25.526800092 +0000 UTC m=+292.600327296" watchObservedRunningTime="2026-11-02 17:28:25.527063426 +0000 UTC m=+292.600590628" Nov 02 17:28:25.659007 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.658972 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qnpf"] Nov 02 17:28:25.663109 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:25.663080 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba9cfaee_932f_42bc_9e42_cc23546474e0.slice/crio-86a5e352f97fd734c6eb3e83406d5d45b733feaf39e2fa9740f28b9a1af5d832 WatchSource:0}: Error finding container 86a5e352f97fd734c6eb3e83406d5d45b733feaf39e2fa9740f28b9a1af5d832: Status 404 returned error can't find the container with id 86a5e352f97fd734c6eb3e83406d5d45b733feaf39e2fa9740f28b9a1af5d832 Nov 02 17:28:25.707847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:25.707754 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:28:25.710406 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:25.710379 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069d21b8_f43f_465f_870f_1da8bfc01136.slice/crio-7ee9ae004e657bb261cb9ff0f90b7787b5a4b64830de84214e7d12e3b54011fb WatchSource:0}: Error finding container 7ee9ae004e657bb261cb9ff0f90b7787b5a4b64830de84214e7d12e3b54011fb: Status 404 returned error can't find the container with id 7ee9ae004e657bb261cb9ff0f90b7787b5a4b64830de84214e7d12e3b54011fb Nov 02 17:28:26.417286 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:26.417250 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" event={"ID":"069d21b8-f43f-465f-870f-1da8bfc01136","Type":"ContainerStarted","Data":"7ee9ae004e657bb261cb9ff0f90b7787b5a4b64830de84214e7d12e3b54011fb"} Nov 02 17:28:26.418347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:26.418320 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" event={"ID":"ba9cfaee-932f-42bc-9e42-cc23546474e0","Type":"ContainerStarted","Data":"86a5e352f97fd734c6eb3e83406d5d45b733feaf39e2fa9740f28b9a1af5d832"} Nov 02 17:28:29.438983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.438942 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" event={"ID":"069d21b8-f43f-465f-870f-1da8bfc01136","Type":"ContainerStarted","Data":"4266d670b6ed80a5c36d558d90c6f34274a19d0a81ed7ea6b46f133166e578b1"} Nov 02 17:28:29.439500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.439045 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:29.441328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.441290 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" event={"ID":"e6b2d4f9-4b40-411f-b365-51d2e171a85a","Type":"ContainerStarted","Data":"d4ed329be27378e6a8bb870a35d088b0ad618de758d3367f212bd695409de31b"} Nov 02 17:28:29.441507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.441395 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:29.442867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.442842 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" event={"ID":"ba9cfaee-932f-42bc-9e42-cc23546474e0","Type":"ContainerStarted","Data":"d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9"} Nov 02 17:28:29.558788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.558705 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" podStartSLOduration=2.6119827129999997 podStartE2EDuration="5.558691105s" totalImagesPullingTime="2.946708392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:29.558163257 +0000 UTC m=+296.631690460" watchObservedRunningTime="2026-11-02 17:28:29.558691105 +0000 UTC m=+296.632218308" Nov 02 17:28:29.672364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.672252 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" podStartSLOduration=1.983017738 podStartE2EDuration="5.672238142s" totalImagesPullingTime="3.689220404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:29.67210114 +0000 UTC m=+296.745628343" watchObservedRunningTime="2026-11-02 17:28:29.672238142 +0000 UTC m=+296.745765344" Nov 02 17:28:29.796332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:29.796279 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" podStartSLOduration=2.383932266 podStartE2EDuration="6.796262653s" totalImagesPullingTime="4.412330387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:29.794803911 +0000 UTC m=+296.868331113" watchObservedRunningTime="2026-11-02 17:28:29.796262653 +0000 UTC m=+296.869789855" Nov 02 17:28:33.096913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.096884 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qnpf"] Nov 02 17:28:33.097333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.097065 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" podUID="ba9cfaee-932f-42bc-9e42-cc23546474e0" containerName="authorino" containerID="cri-o://d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9" gracePeriod=30 Nov 02 17:28:33.137143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.137118 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:28:33.138102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.138081 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:28:33.350468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.350398 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:33.380427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.380397 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr\" (UniqueName: \"kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr\") pod \"ba9cfaee-932f-42bc-9e42-cc23546474e0\" (UID: \"ba9cfaee-932f-42bc-9e42-cc23546474e0\") " Nov 02 17:28:33.382684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.382656 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr" pod "ba9cfaee-932f-42bc-9e42-cc23546474e0" (UID: "ba9cfaee-932f-42bc-9e42-cc23546474e0"). InnerVolumeSpecName "kube-api-access-92zcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:28:33.461764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.461712 5647 generic.go:356] "Generic (PLEG): container finished" podID="ba9cfaee-932f-42bc-9e42-cc23546474e0" containerID="d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9" exitCode=0 Nov 02 17:28:33.461937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.461776 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" Nov 02 17:28:33.461937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.461799 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" event={"ID":"ba9cfaee-932f-42bc-9e42-cc23546474e0","Type":"ContainerDied","Data":"d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9"} Nov 02 17:28:33.461937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.461844 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4qnpf" event={"ID":"ba9cfaee-932f-42bc-9e42-cc23546474e0","Type":"ContainerDied","Data":"86a5e352f97fd734c6eb3e83406d5d45b733feaf39e2fa9740f28b9a1af5d832"} Nov 02 17:28:33.461937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.461862 5647 scope.go:122] "RemoveContainer" containerID="d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9" Nov 02 17:28:33.471666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.471644 5647 scope.go:122] "RemoveContainer" containerID="d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9" Nov 02 17:28:33.472092 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:28:33.472039 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9\": container with ID starting with d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9 not found: ID does not exist" containerID="d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9" Nov 02 17:28:33.472240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.472108 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9"} err="failed to get container status \"d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9\": rpc error: code = NotFound desc = could not find container \"d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9\": container with ID starting with d6219f55680fac5dc0e988ba001d9d934e21dc47edc4995a283a02e73e2959d9 not found: ID does not exist" Nov 02 17:28:33.481840 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.481771 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92zcr\" (UniqueName: \"kubernetes.io/projected/ba9cfaee-932f-42bc-9e42-cc23546474e0-kube-api-access-92zcr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:28:33.600147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.600115 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qnpf"] Nov 02 17:28:33.605597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:33.605534 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4qnpf"] Nov 02 17:28:35.079404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:35.079368 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9cfaee-932f-42bc-9e42-cc23546474e0" path="/var/lib/kubelet/pods/ba9cfaee-932f-42bc-9e42-cc23546474e0/volumes" Nov 02 17:28:35.954523 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:28:35.954473 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:28:40.449622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:40.449591 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-dzrt7" Nov 02 17:28:40.450090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:40.449666 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:40.600951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:40.600911 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:40.601158 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:40.601121 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" podUID="e6b2d4f9-4b40-411f-b365-51d2e171a85a" containerName="limitador" containerID="cri-o://d4ed329be27378e6a8bb870a35d088b0ad618de758d3367f212bd695409de31b" gracePeriod=30 Nov 02 17:28:41.498655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.498626 5647 generic.go:356] "Generic (PLEG): container finished" podID="e6b2d4f9-4b40-411f-b365-51d2e171a85a" containerID="d4ed329be27378e6a8bb870a35d088b0ad618de758d3367f212bd695409de31b" exitCode=0 Nov 02 17:28:41.499057 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.498692 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" event={"ID":"e6b2d4f9-4b40-411f-b365-51d2e171a85a","Type":"ContainerDied","Data":"d4ed329be27378e6a8bb870a35d088b0ad618de758d3367f212bd695409de31b"} Nov 02 17:28:41.554909 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.554888 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:41.652635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.652598 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7\" (UniqueName: \"kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7\") pod \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " Nov 02 17:28:41.652858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.652688 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file\" (UniqueName: \"kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file\") pod \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\" (UID: \"e6b2d4f9-4b40-411f-b365-51d2e171a85a\") " Nov 02 17:28:41.653086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.653058 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file" pod "e6b2d4f9-4b40-411f-b365-51d2e171a85a" (UID: "e6b2d4f9-4b40-411f-b365-51d2e171a85a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:28:41.655206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.655169 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7" pod "e6b2d4f9-4b40-411f-b365-51d2e171a85a" (UID: "e6b2d4f9-4b40-411f-b365-51d2e171a85a"). InnerVolumeSpecName "kube-api-access-2g9l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:28:41.753986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.753907 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g9l7\" (UniqueName: \"kubernetes.io/projected/e6b2d4f9-4b40-411f-b365-51d2e171a85a-kube-api-access-2g9l7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:28:41.753986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:41.753933 5647 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e6b2d4f9-4b40-411f-b365-51d2e171a85a-config-file\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:28:42.503990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:42.503939 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" event={"ID":"e6b2d4f9-4b40-411f-b365-51d2e171a85a","Type":"ContainerDied","Data":"2223c21bbe226662114379d7bfde765a26a00f6facd9e380cafae725655f645b"} Nov 02 17:28:42.503990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:42.503968 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9rhdw" Nov 02 17:28:42.504461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:42.504027 5647 scope.go:122] "RemoveContainer" containerID="d4ed329be27378e6a8bb870a35d088b0ad618de758d3367f212bd695409de31b" Nov 02 17:28:42.673573 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:42.673538 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:42.724254 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:42.724216 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9rhdw"] Nov 02 17:28:43.080154 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:43.080123 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b2d4f9-4b40-411f-b365-51d2e171a85a" path="/var/lib/kubelet/pods/e6b2d4f9-4b40-411f-b365-51d2e171a85a/volumes" Nov 02 17:28:46.695716 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.695684 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl"] Nov 02 17:28:46.696174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.696088 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ba9cfaee-932f-42bc-9e42-cc23546474e0" containerName="authorino" Nov 02 17:28:46.696174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.696101 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9cfaee-932f-42bc-9e42-cc23546474e0" containerName="authorino" Nov 02 17:28:46.696174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.696122 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e6b2d4f9-4b40-411f-b365-51d2e171a85a" containerName="limitador" Nov 02 17:28:46.696174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.696128 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b2d4f9-4b40-411f-b365-51d2e171a85a" containerName="limitador" Nov 02 17:28:46.696174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.696178 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b2d4f9-4b40-411f-b365-51d2e171a85a" containerName="limitador" Nov 02 17:28:46.696360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.696190 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9cfaee-932f-42bc-9e42-cc23546474e0" containerName="authorino" Nov 02 17:28:46.699117 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.699099 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:46.754428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.754402 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 17:28:46.803207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.803171 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c1cb0815-f24d-4ace-a544-ef0c90607758-data\") pod \"postgres-7cdbf6dc44-rblzl\" (UID: \"c1cb0815-f24d-4ace-a544-ef0c90607758\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:46.803371 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.803225 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdzcf\" (UniqueName: \"kubernetes.io/projected/c1cb0815-f24d-4ace-a544-ef0c90607758-kube-api-access-bdzcf\") pod \"postgres-7cdbf6dc44-rblzl\" (UID: \"c1cb0815-f24d-4ace-a544-ef0c90607758\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:46.805285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.805249 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cspm9\"" Nov 02 17:28:46.805285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.805249 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 17:28:46.805418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.805334 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 17:28:46.806357 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.806339 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl"] Nov 02 17:28:46.904234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.904201 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdzcf\" (UniqueName: \"kubernetes.io/projected/c1cb0815-f24d-4ace-a544-ef0c90607758-kube-api-access-bdzcf\") pod \"postgres-7cdbf6dc44-rblzl\" (UID: \"c1cb0815-f24d-4ace-a544-ef0c90607758\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:46.904410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.904290 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c1cb0815-f24d-4ace-a544-ef0c90607758-data\") pod \"postgres-7cdbf6dc44-rblzl\" (UID: \"c1cb0815-f24d-4ace-a544-ef0c90607758\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:46.904616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.904599 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c1cb0815-f24d-4ace-a544-ef0c90607758-data\") pod \"postgres-7cdbf6dc44-rblzl\" (UID: \"c1cb0815-f24d-4ace-a544-ef0c90607758\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:46.971048 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:46.970969 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdzcf\" (UniqueName: \"kubernetes.io/projected/c1cb0815-f24d-4ace-a544-ef0c90607758-kube-api-access-bdzcf\") pod \"postgres-7cdbf6dc44-rblzl\" (UID: \"c1cb0815-f24d-4ace-a544-ef0c90607758\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:47.014221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:47.014172 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:47.386682 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:47.386649 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl"] Nov 02 17:28:47.388506 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:28:47.388475 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cb0815_f24d_4ace_a544_ef0c90607758.slice/crio-c7b35fa9e911214c2fcc7a6b74413023bedc33e8915e5442be7c847e4838fed9 WatchSource:0}: Error finding container c7b35fa9e911214c2fcc7a6b74413023bedc33e8915e5442be7c847e4838fed9: Status 404 returned error can't find the container with id c7b35fa9e911214c2fcc7a6b74413023bedc33e8915e5442be7c847e4838fed9 Nov 02 17:28:47.389989 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:47.389965 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:28:47.536323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:47.536279 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" event={"ID":"c1cb0815-f24d-4ace-a544-ef0c90607758","Type":"ContainerStarted","Data":"c7b35fa9e911214c2fcc7a6b74413023bedc33e8915e5442be7c847e4838fed9"} Nov 02 17:28:55.575548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:55.575510 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" event={"ID":"c1cb0815-f24d-4ace-a544-ef0c90607758","Type":"ContainerStarted","Data":"ddf23e5ba2b81ec54b59a2ea45c516b92975e868ccf57857ed45884d30bff641"} Nov 02 17:28:55.576034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:55.575644 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:28:55.700604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:28:55.700536 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" podStartSLOduration=2.12385404 podStartE2EDuration="9.700520127s" totalImagesPullingTime="7.576666087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:28:55.699655833 +0000 UTC m=+322.773183036" watchObservedRunningTime="2026-11-02 17:28:55.700520127 +0000 UTC m=+322.774047324" Nov 02 17:29:01.609822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:01.609791 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rblzl" Nov 02 17:29:05.946409 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:05.946369 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:29:08.822099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.822065 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66f466b5d6-mmpsl"] Nov 02 17:29:08.826689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.826663 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:08.881115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.881075 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 17:29:08.914558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.914511 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:08.914792 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.914649 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:08.914858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.914787 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc85\" (UniqueName: \"kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:08.932292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.932258 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 17:29:08.933161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.932616 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 17:29:08.933161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.932702 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 17:29:08.933161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.933036 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 17:29:08.933161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.933088 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tk2kj\"" Nov 02 17:29:08.934702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:08.934675 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66f466b5d6-mmpsl"] Nov 02 17:29:09.015436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.015395 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc85\" (UniqueName: \"kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.015654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.015448 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.015654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.015515 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.018409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.018382 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.018536 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.018425 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.076578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.076464 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc85\" (UniqueName: \"kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85\") pod \"maas-controller-66f466b5d6-mmpsl\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.142229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.142181 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:09.406771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.406724 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-lqt67"] Nov 02 17:29:09.409400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.409378 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.470777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.470727 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-lqt67"] Nov 02 17:29:09.520168 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.520136 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.520341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.520206 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.520341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.520299 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p62j\" (UniqueName: \"kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.529515 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.529487 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66f466b5d6-mmpsl"] Nov 02 17:29:09.530884 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:29:09.530855 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15faf9e3_ee3d_4760_861e_88cb51dc7953.slice/crio-0b2f557d4a3213f802cffa6f69699b0fee0853a0898d077ce3694d0e92123599 WatchSource:0}: Error finding container 0b2f557d4a3213f802cffa6f69699b0fee0853a0898d077ce3694d0e92123599: Status 404 returned error can't find the container with id 0b2f557d4a3213f802cffa6f69699b0fee0853a0898d077ce3694d0e92123599 Nov 02 17:29:09.620878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.620841 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.621054 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.620898 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.621054 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.620928 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p62j\" (UniqueName: \"kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.623445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.623416 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.623558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.623461 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.637230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.637203 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" event={"ID":"15faf9e3-ee3d-4760-861e-88cb51dc7953","Type":"ContainerStarted","Data":"0b2f557d4a3213f802cffa6f69699b0fee0853a0898d077ce3694d0e92123599"} Nov 02 17:29:09.679496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.679425 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p62j\" (UniqueName: \"kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j\") pod \"maas-controller-7fd65b9f5-lqt67\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:09.721330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:09.721299 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:10.099174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:10.099142 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-lqt67"] Nov 02 17:29:10.102150 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:29:10.102118 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9e8aee_a1e8_47ba_95fd_a7a5067d9367.slice/crio-385c2eb65f792aa1d69bccd4a574e2712d2b58b292ee5bef4c7d1c12f933d04d WatchSource:0}: Error finding container 385c2eb65f792aa1d69bccd4a574e2712d2b58b292ee5bef4c7d1c12f933d04d: Status 404 returned error can't find the container with id 385c2eb65f792aa1d69bccd4a574e2712d2b58b292ee5bef4c7d1c12f933d04d Nov 02 17:29:10.643545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:10.643505 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" event={"ID":"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367","Type":"ContainerStarted","Data":"385c2eb65f792aa1d69bccd4a574e2712d2b58b292ee5bef4c7d1c12f933d04d"} Nov 02 17:29:14.666391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:14.666267 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" event={"ID":"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367","Type":"ContainerStarted","Data":"72780a2c28595b9f97e42fdb8b1ba15a1d76fb1c9552c287471e1b817f24de45"} Nov 02 17:29:14.666391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:14.666367 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:14.863506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:14.863401 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" podStartSLOduration=1.36639551 podStartE2EDuration="5.863388611s" totalImagesPullingTime="4.496993101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:14.861920319 +0000 UTC m=+341.935447522" watchObservedRunningTime="2026-11-02 17:29:14.863388611 +0000 UTC m=+341.936915814" Nov 02 17:29:15.673157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:15.673114 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" event={"ID":"15faf9e3-ee3d-4760-861e-88cb51dc7953","Type":"ContainerStarted","Data":"14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32"} Nov 02 17:29:15.793524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:15.793465 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" podStartSLOduration=2.662549476 podStartE2EDuration="7.79344891s" totalImagesPullingTime="5.130899434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:15.792414097 +0000 UTC m=+342.865941363" watchObservedRunningTime="2026-11-02 17:29:15.79344891 +0000 UTC m=+342.866976107" Nov 02 17:29:16.678576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:16.678530 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:18.080905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.080873 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969"] Nov 02 17:29:18.084507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.084487 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.101633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.101605 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.101872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.101640 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.101872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.101657 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpt9r\" (UniqueName: \"kubernetes.io/projected/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-kube-api-access-mpt9r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.101872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.101856 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.140208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.140178 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 17:29:18.140354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.140225 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 17:29:18.189977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.189952 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 17:29:18.190336 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.190321 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nrncr\"" Nov 02 17:29:18.191506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.191479 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969"] Nov 02 17:29:18.203028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.202997 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.203115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.203035 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.203115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.203090 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.203198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.203122 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpt9r\" (UniqueName: \"kubernetes.io/projected/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-kube-api-access-mpt9r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.203198 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:18.203173 5647 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 17:29:18.203257 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:18.203250 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-proxy-source podName:6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:18.703227771 +0000 UTC m=+345.776754970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-b6969" (UID: "6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b") : configmap "usage-tenancy-proxy-source" not found Nov 02 17:29:18.203297 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:18.203263 5647 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 17:29:18.203332 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:18.203307 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-tls-cert podName:6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b nodeName:}" failed. No retries permitted until 2026-11-02 17:29:18.703294966 +0000 UTC m=+345.776822146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-b6969" (UID: "6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 17:29:18.203694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.203678 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.264998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.264963 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpt9r\" (UniqueName: \"kubernetes.io/projected/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-kube-api-access-mpt9r\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.708266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.708231 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.708453 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.708276 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.709143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.709088 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.710838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.710809 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-b6969\" (UID: \"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:18.997254 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:18.997162 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:19.372972 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:19.372945 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969"] Nov 02 17:29:19.375507 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:29:19.375479 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a605b3e_d7f1_4af2_89aa_e0063bfb2c7b.slice/crio-07e8aa02907184c261b5e99db267bf610dd5031fe68a62a46c013aa48ce03c24 WatchSource:0}: Error finding container 07e8aa02907184c261b5e99db267bf610dd5031fe68a62a46c013aa48ce03c24: Status 404 returned error can't find the container with id 07e8aa02907184c261b5e99db267bf610dd5031fe68a62a46c013aa48ce03c24 Nov 02 17:29:19.693965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:19.693879 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" event={"ID":"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b","Type":"ContainerStarted","Data":"07e8aa02907184c261b5e99db267bf610dd5031fe68a62a46c013aa48ce03c24"} Nov 02 17:29:25.675822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:25.675759 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 17:29:25.857021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:25.856933 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66f466b5d6-mmpsl"] Nov 02 17:29:25.857388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:25.857356 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" podUID="15faf9e3-ee3d-4760-861e-88cb51dc7953" containerName="manager" containerID="cri-o://14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32" gracePeriod=30 Nov 02 17:29:25.859359 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:25.859273 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:26.148657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.148627 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:26.188035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.187999 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs\") pod \"15faf9e3-ee3d-4760-861e-88cb51dc7953\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " Nov 02 17:29:26.188232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.188113 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85\" (UniqueName: \"kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85\") pod \"15faf9e3-ee3d-4760-861e-88cb51dc7953\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " Nov 02 17:29:26.188232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.188186 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert\") pod \"15faf9e3-ee3d-4760-861e-88cb51dc7953\" (UID: \"15faf9e3-ee3d-4760-861e-88cb51dc7953\") " Nov 02 17:29:26.191824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.191791 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs" pod "15faf9e3-ee3d-4760-861e-88cb51dc7953" (UID: "15faf9e3-ee3d-4760-861e-88cb51dc7953"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:29:26.194099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.193810 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert" pod "15faf9e3-ee3d-4760-861e-88cb51dc7953" (UID: "15faf9e3-ee3d-4760-861e-88cb51dc7953"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:29:26.194099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.194072 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85" pod "15faf9e3-ee3d-4760-861e-88cb51dc7953" (UID: "15faf9e3-ee3d-4760-861e-88cb51dc7953"). InnerVolumeSpecName "kube-api-access-6tc85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:29:26.289102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.289009 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tc85\" (UniqueName: \"kubernetes.io/projected/15faf9e3-ee3d-4760-861e-88cb51dc7953-kube-api-access-6tc85\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:29:26.289102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.289046 5647 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-webhook-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:29:26.289102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.289063 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15faf9e3-ee3d-4760-861e-88cb51dc7953-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:29:26.732451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.732410 5647 generic.go:356] "Generic (PLEG): container finished" podID="15faf9e3-ee3d-4760-861e-88cb51dc7953" containerID="14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32" exitCode=0 Nov 02 17:29:26.732997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.732540 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" event={"ID":"15faf9e3-ee3d-4760-861e-88cb51dc7953","Type":"ContainerDied","Data":"14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32"} Nov 02 17:29:26.732997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.732599 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" event={"ID":"15faf9e3-ee3d-4760-861e-88cb51dc7953","Type":"ContainerDied","Data":"0b2f557d4a3213f802cffa6f69699b0fee0853a0898d077ce3694d0e92123599"} Nov 02 17:29:26.732997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.732622 5647 scope.go:122] "RemoveContainer" containerID="14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32" Nov 02 17:29:26.732997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.732792 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66f466b5d6-mmpsl" Nov 02 17:29:26.748211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.748184 5647 scope.go:122] "RemoveContainer" containerID="14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32" Nov 02 17:29:26.749041 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:26.749007 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32\": container with ID starting with 14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32 not found: ID does not exist" containerID="14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32" Nov 02 17:29:26.749153 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.749053 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32"} err="failed to get container status \"14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32\": rpc error: code = NotFound desc = could not find container \"14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32\": container with ID starting with 14f79134d7b1246d97855072a2aa41ac2f5005a17b039ae4161bb744b9ce4d32 not found: ID does not exist" Nov 02 17:29:26.910327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.910291 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66f466b5d6-mmpsl"] Nov 02 17:29:26.961268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:26.961232 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66f466b5d6-mmpsl"] Nov 02 17:29:27.079939 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:27.079864 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15faf9e3-ee3d-4760-861e-88cb51dc7953" path="/var/lib/kubelet/pods/15faf9e3-ee3d-4760-861e-88cb51dc7953/volumes" Nov 02 17:29:33.769942 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:33.769900 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" event={"ID":"6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b","Type":"ContainerStarted","Data":"8f9869339dee214079541dba94151b97b83a6229b04605f2bb842965d76d06b2"} Nov 02 17:29:33.770426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:33.770029 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:33.888448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:33.888378 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" podStartSLOduration=2.001403436 podStartE2EDuration="15.888364149s" totalImagesPullingTime="13.886960713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:33.887328183 +0000 UTC m=+360.960855419" watchObservedRunningTime="2026-11-02 17:29:33.888364149 +0000 UTC m=+360.961891398" Nov 02 17:29:35.913641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.913605 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg"] Nov 02 17:29:35.914057 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.914016 5647 cpu_manager.go:414] "containerMap: removing container" podUID="15faf9e3-ee3d-4760-861e-88cb51dc7953" containerName="manager" Nov 02 17:29:35.914057 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.914029 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15faf9e3-ee3d-4760-861e-88cb51dc7953" containerName="manager" Nov 02 17:29:35.914132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.914120 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="15faf9e3-ee3d-4760-861e-88cb51dc7953" containerName="manager" Nov 02 17:29:35.917373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.917357 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:35.947072 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:29:35.947028 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:29:35.970877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.970853 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 17:29:35.971123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.971104 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 17:29:35.971167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.971111 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 17:29:35.971201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.971175 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gcxmh\"" Nov 02 17:29:35.974794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:35.974772 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 17:29:36.010980 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.010936 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.010980 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.010977 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5nb\" (UniqueName: \"kubernetes.io/projected/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-kube-api-access-fk5nb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.011202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.011100 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.011202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.011131 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.011202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.011174 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.011301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.011280 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.011337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.011306 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.023239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.023214 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg"] Nov 02 17:29:36.112345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112297 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.112345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112343 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.112580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112377 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.112580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112408 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.112580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112428 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.112580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112525 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.112580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.112564 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5nb\" (UniqueName: \"kubernetes.io/projected/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-kube-api-access-fk5nb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.113326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.113303 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.114978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.114953 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.115123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.115096 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.115217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.115198 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.115381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.115363 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.177457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.177386 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.177594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.177565 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5nb\" (UniqueName: \"kubernetes.io/projected/3d22bc9a-c7e3-47c2-acef-81df6d69ffa9-kube-api-access-fk5nb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-svpxg\" (UID: \"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.229935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.229901 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:36.658271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.658234 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg"] Nov 02 17:29:36.694993 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:29:36.694964 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d22bc9a_c7e3_47c2_acef_81df6d69ffa9.slice/crio-bfdde9fabeaf855f966659096c402d0e0ca6f240eee4eee14c9975110311dbde WatchSource:0}: Error finding container bfdde9fabeaf855f966659096c402d0e0ca6f240eee4eee14c9975110311dbde: Status 404 returned error can't find the container with id bfdde9fabeaf855f966659096c402d0e0ca6f240eee4eee14c9975110311dbde Nov 02 17:29:36.785614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:36.785579 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" event={"ID":"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9","Type":"ContainerStarted","Data":"bfdde9fabeaf855f966659096c402d0e0ca6f240eee4eee14c9975110311dbde"} Nov 02 17:29:40.192553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:40.192522 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:29:40.192816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:40.192587 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:29:40.806964 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:40.806923 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" event={"ID":"3d22bc9a-c7e3-47c2-acef-81df6d69ffa9","Type":"ContainerStarted","Data":"8c8619e779ca15d73257b1205c277b81aea217dd35cd2e6b4108fb56658e9d79"} Nov 02 17:29:40.807208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:40.806987 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:41.029472 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:41.029407 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" podStartSLOduration=2.534788889 podStartE2EDuration="6.029388404s" totalImagesPullingTime="3.494599515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:40.977871593 +0000 UTC m=+368.051398796" watchObservedRunningTime="2026-11-02 17:29:41.029388404 +0000 UTC m=+368.102915609" Nov 02 17:29:41.813140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:41.813068 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-svpxg" Nov 02 17:29:44.780428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:44.780398 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-b6969" Nov 02 17:29:57.331141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:29:57.331100 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:30:05.946111 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:30:05.946070 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:30:35.946916 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:30:35.946867 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:31:05.946759 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:31:05.946693 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:31:35.947771 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:31:35.947669 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:32:04.325870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:04.325837 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:32:05.946598 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:32:05.946552 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:32:11.661120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:11.661087 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:32:18.484608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.484570 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd"] Nov 02 17:32:18.488182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.488162 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.544460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.544427 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 17:32:18.544628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.544488 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 17:32:18.595213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.595169 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 17:32:18.595213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.595196 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5r2ch\"" Nov 02 17:32:18.596397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.596372 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd"] Nov 02 17:32:18.649582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.649550 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhpf\" (UniqueName: \"kubernetes.io/projected/2a224289-7a58-42a9-ae19-14b3050f9099-kube-api-access-gzhpf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.649772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.649605 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.649772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.649703 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a224289-7a58-42a9-ae19-14b3050f9099-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.649899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.649825 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.649961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.649893 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.649961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.649932 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.750999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.750917 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhpf\" (UniqueName: \"kubernetes.io/projected/2a224289-7a58-42a9-ae19-14b3050f9099-kube-api-access-gzhpf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.750999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.750985 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751016 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a224289-7a58-42a9-ae19-14b3050f9099-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751158 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751349 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751233 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751349 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751288 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751444 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751598 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.751692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.751641 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.753643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.753622 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a224289-7a58-42a9-ae19-14b3050f9099-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.753844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.753827 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a224289-7a58-42a9-ae19-14b3050f9099-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:18.810682 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:18.810640 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhpf\" (UniqueName: \"kubernetes.io/projected/2a224289-7a58-42a9-ae19-14b3050f9099-kube-api-access-gzhpf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd\" (UID: \"2a224289-7a58-42a9-ae19-14b3050f9099\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:19.102194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:19.102156 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:19.478767 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:19.478676 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd"] Nov 02 17:32:19.481156 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:32:19.481125 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a224289_7a58_42a9_ae19_14b3050f9099.slice/crio-29a1d5a0df5445d5975013f3a2741669b229593f2fab65edd6d4b6521d13b849 WatchSource:0}: Error finding container 29a1d5a0df5445d5975013f3a2741669b229593f2fab65edd6d4b6521d13b849: Status 404 returned error can't find the container with id 29a1d5a0df5445d5975013f3a2741669b229593f2fab65edd6d4b6521d13b849 Nov 02 17:32:19.501471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:19.501433 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" event={"ID":"2a224289-7a58-42a9-ae19-14b3050f9099","Type":"ContainerStarted","Data":"29a1d5a0df5445d5975013f3a2741669b229593f2fab65edd6d4b6521d13b849"} Nov 02 17:32:21.516770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:21.516663 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" event={"ID":"2a224289-7a58-42a9-ae19-14b3050f9099","Type":"ContainerStarted","Data":"de7a0b416bf4ce69753e9fab36e58931f817816dbb961f9733a699b380eb00b7"} Nov 02 17:32:21.635260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:21.635210 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" podStartSLOduration=1.929720764 podStartE2EDuration="3.635196931s" totalImagesPullingTime="1.705476167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:21.633998618 +0000 UTC m=+528.707525820" watchObservedRunningTime="2026-11-02 17:32:21.635196931 +0000 UTC m=+528.708724134" Nov 02 17:32:22.102994 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.102954 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:22.823638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.823607 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hd9vk"] Nov 02 17:32:22.827426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.827407 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:22.883082 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.883052 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 02 17:32:22.935309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.935278 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hd9vk"] Nov 02 17:32:22.994474 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.994435 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c622747f-8ec4-45a4-85af-c79f65ba4d13-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:22.994644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.994480 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:22.994644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.994507 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:22.994644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.994624 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-home\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:22.994820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.994720 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8st47\" (UniqueName: \"kubernetes.io/projected/c622747f-8ec4-45a4-85af-c79f65ba4d13-kube-api-access-8st47\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:22.994820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:22.994790 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095225 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8st47\" (UniqueName: \"kubernetes.io/projected/c622747f-8ec4-45a4-85af-c79f65ba4d13-kube-api-access-8st47\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095274 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095316 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c622747f-8ec4-45a4-85af-c79f65ba4d13-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095339 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095358 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095377 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-home\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095782 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095821 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-home\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.095959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.095886 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.097907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.097881 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c622747f-8ec4-45a4-85af-c79f65ba4d13-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.098022 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.097998 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c622747f-8ec4-45a4-85af-c79f65ba4d13-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.153324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.153290 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8st47\" (UniqueName: \"kubernetes.io/projected/c622747f-8ec4-45a4-85af-c79f65ba4d13-kube-api-access-8st47\") pod \"echo-simulator-kserve-64b6f8f96c-hd9vk\" (UID: \"c622747f-8ec4-45a4-85af-c79f65ba4d13\") " pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.440349 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.440266 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:23.820790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:23.820768 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-hd9vk"] Nov 02 17:32:23.823303 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:32:23.823274 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc622747f_8ec4_45a4_85af_c79f65ba4d13.slice/crio-6b3b2d33c7985d3c93b51c363d5ca19d67a3a1f3a503058507e4285f77acb20d WatchSource:0}: Error finding container 6b3b2d33c7985d3c93b51c363d5ca19d67a3a1f3a503058507e4285f77acb20d: Status 404 returned error can't find the container with id 6b3b2d33c7985d3c93b51c363d5ca19d67a3a1f3a503058507e4285f77acb20d Nov 02 17:32:24.532423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:24.532387 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" event={"ID":"c622747f-8ec4-45a4-85af-c79f65ba4d13","Type":"ContainerStarted","Data":"6b3b2d33c7985d3c93b51c363d5ca19d67a3a1f3a503058507e4285f77acb20d"} Nov 02 17:32:26.055494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:26.055454 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:32:26.543485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:26.543449 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" event={"ID":"c622747f-8ec4-45a4-85af-c79f65ba4d13","Type":"ContainerStarted","Data":"80555206cc4f5ed45076c6e09f5d91abe01cdf784b93dc4147761ec17af62967"} Nov 02 17:32:26.678983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:26.678923 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" podStartSLOduration=2.212833419 podStartE2EDuration="4.678909027s" totalImagesPullingTime="2.466075608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:26.677243008 +0000 UTC m=+533.750770222" watchObservedRunningTime="2026-11-02 17:32:26.678909027 +0000 UTC m=+533.752436227" Nov 02 17:32:27.440914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:27.440879 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:29.102384 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:29.102342 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:29.115146 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:29.115122 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:29.568302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:29.568225 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd" Nov 02 17:32:33.441101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.441060 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:33.453903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.453872 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:33.586435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.586407 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-hd9vk" Nov 02 17:32:33.788953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.788863 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp"] Nov 02 17:32:33.792538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.792521 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.802326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.802303 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz"] Nov 02 17:32:33.806362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.806341 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.850643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.850610 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp"] Nov 02 17:32:33.850866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.850846 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\"" Nov 02 17:32:33.895263 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.895232 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz"] Nov 02 17:32:33.899635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899609 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.899803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899663 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.899803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899699 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.899803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899746 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.899803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899775 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.899983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899862 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.899983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899906 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.899983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899939 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.899983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.899961 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.900191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900015 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.900245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900168 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6s9m\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.900245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900226 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.900352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900255 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.900352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900337 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.900449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900363 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.900449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900410 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdtwc\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:33.900449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900436 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:33.900596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:33.900501 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.001872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.001835 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6s9m\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.001872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.001878 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.001897 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.001993 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002010 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002038 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdtwc\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002053 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002080 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002108 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002157 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002206 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002247 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002285 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002311 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002338 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002373 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002412 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002418 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002441 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.002561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002479 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.003335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002773 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.003335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002893 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.003335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.002923 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.003335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.003086 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.003335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.003245 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.003602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.003337 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.003602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.003473 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.003602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.003491 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.004584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.004556 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.004797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.004779 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.004979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.004960 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.005595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.005578 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.060016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.059927 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.060159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.060045 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdtwc\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-767tz\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.060159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.060084 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.060244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.060187 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6s9m\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-wb5mp\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.106386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.106346 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:34.120161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.120128 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:34.536203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.536176 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp"] Nov 02 17:32:34.537533 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.537510 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz"] Nov 02 17:32:34.537856 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:32:34.537827 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02cefeb_4a1f_41b8_bf36_50ccb45f3333.slice/crio-f6acd838596b3e96ef1de167ec3ce6ec978e3621cbc917e5c09f611306c1e8d0 WatchSource:0}: Error finding container f6acd838596b3e96ef1de167ec3ce6ec978e3621cbc917e5c09f611306c1e8d0: Status 404 returned error can't find the container with id f6acd838596b3e96ef1de167ec3ce6ec978e3621cbc917e5c09f611306c1e8d0 Nov 02 17:32:34.575480 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:32:34.575441 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d78f16e_044f_4b75_aeff_3eec744367ad.slice/crio-6e2b2bda773f74743f896b4aa20f69a31e1fc0ef534b4d692318974899ce5e4b WatchSource:0}: Error finding container 6e2b2bda773f74743f896b4aa20f69a31e1fc0ef534b4d692318974899ce5e4b: Status 404 returned error can't find the container with id 6e2b2bda773f74743f896b4aa20f69a31e1fc0ef534b4d692318974899ce5e4b Nov 02 17:32:34.581264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:34.581234 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" event={"ID":"e02cefeb-4a1f-41b8-bf36-50ccb45f3333","Type":"ContainerStarted","Data":"f6acd838596b3e96ef1de167ec3ce6ec978e3621cbc917e5c09f611306c1e8d0"} Nov 02 17:32:35.588091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:35.588057 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" event={"ID":"4d78f16e-044f-4b75-aeff-3eec744367ad","Type":"ContainerStarted","Data":"6e2b2bda773f74743f896b4aa20f69a31e1fc0ef534b4d692318974899ce5e4b"} Nov 02 17:32:35.947214 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:32:35.947075 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:32:38.867614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:38.867577 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:32:38.867975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:38.867651 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:32:38.867975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:38.867679 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:32:38.974845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:38.974809 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:32:38.974968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:38.974897 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:32:38.974968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:38.974943 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:32:39.654080 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:39.654035 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" event={"ID":"4d78f16e-044f-4b75-aeff-3eec744367ad","Type":"ContainerStarted","Data":"b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9"} Nov 02 17:32:39.656051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:39.656016 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" event={"ID":"e02cefeb-4a1f-41b8-bf36-50ccb45f3333","Type":"ContainerStarted","Data":"eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82"} Nov 02 17:32:39.892598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:39.892543 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podStartSLOduration=2.496530683 podStartE2EDuration="6.89252847s" totalImagesPullingTime="4.395997787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:39.891956055 +0000 UTC m=+546.965483258" watchObservedRunningTime="2026-11-02 17:32:39.89252847 +0000 UTC m=+546.966055672" Nov 02 17:32:40.025816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.025237 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podStartSLOduration=2.6983707040000002 podStartE2EDuration="7.025215912s" totalImagesPullingTime="4.326845208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:32:40.024296857 +0000 UTC m=+547.097824061" watchObservedRunningTime="2026-11-02 17:32:40.025215912 +0000 UTC m=+547.098743118" Nov 02 17:32:40.108601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.108516 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:40.122380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.116891 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:40.122380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.116980 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:40.122380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.121149 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:40.128760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.124885 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:40.128760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:40.124960 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:41.111238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:41.111199 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:41.111864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:41.111832 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:41.124154 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:41.124121 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:41.124377 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:41.124351 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:42.116760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:42.113262 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:42.116760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:42.113332 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:42.121354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:42.121311 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:42.121634 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:42.121589 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:43.127759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:43.125174 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:43.127759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:43.125239 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:43.127759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:43.126483 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:43.127759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:43.126532 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:43.341761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:43.340968 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:32:44.109761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:44.107281 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:44.109761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:44.107366 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:44.128337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:44.128299 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:44.129011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:44.128962 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:45.106696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:45.106661 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:45.106929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:45.106715 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:45.120478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:45.120442 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:45.120674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:45.120492 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:46.107815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:46.107772 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Nov 02 17:32:46.108363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:46.107828 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Nov 02 17:32:46.123638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:46.123601 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 02 17:32:46.123845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:46.123670 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 02 17:32:47.431470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:47.431432 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:47.432270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:47.432207 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:47.433393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:47.433369 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:32:47.497179 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:47.497149 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:47.497522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:47.497492 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:32:47.498393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:32:47.498373 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 17:33:05.946021 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:33:05.945982 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:33:32.032004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.031966 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf"] Nov 02 17:33:32.034635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.034617 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.088174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.088135 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-aa5b913c-openshift-default-dockercfg-d5v7x\"" Nov 02 17:33:32.140767 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.140720 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf"] Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225033 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225080 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225156 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrfx\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225194 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225277 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225299 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225349 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225399 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.231848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.225481 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.325955 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326025 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326044 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326072 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326103 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326153 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326182 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326205 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326315 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrfx\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326516 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326571 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326645 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.326927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.326906 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.327131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.327110 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.328678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.328659 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.329274 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.329254 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.348496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.348469 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf"] Nov 02 17:33:32.350992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.350974 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.384825 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.384788 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.385468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.385438 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrfx\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx\") pod \"e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.411514 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.411484 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf"] Nov 02 17:33:32.455565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.455533 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-97ba043b-openshift-default-dockercfg-c9rrg\"" Nov 02 17:33:32.528791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.528714 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.528987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.528848 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.528987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.528906 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5s4k\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.528987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.528947 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.529108 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.529049 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.529108 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.529087 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.529211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.529192 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.529242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.529225 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.529292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.529246 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630181 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630220 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630270 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630300 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630347 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5s4k\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630383 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630416 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630451 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630524 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.630790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630770 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.631195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630854 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.631195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630928 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.631195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.630992 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.631195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.631136 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.632850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.632821 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.633071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.633053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.646202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.646182 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:32.653978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.653951 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4"] Nov 02 17:33:32.656400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.656385 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.692093 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.692064 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.692294 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.692243 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5s4k\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.709702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.709654 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cc12ac52-gw-openshift-default-dockercfg-tvvbc\"" Nov 02 17:33:32.720214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.720121 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4"] Nov 02 17:33:32.742469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.742430 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.742717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.742684 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.742941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.742781 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.742941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.742887 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.743016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.742973 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57tj\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.743052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.743031 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.743766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.743122 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.743766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.743176 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.743766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.743262 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.843996 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844088 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844118 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844164 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844189 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844223 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57tj\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844257 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844405 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844459 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844663 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844766 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.844852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.844799 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.845032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.845010 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.845349 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.845327 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.846608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.846590 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.846762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.846718 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.902600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.902516 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.902872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.902855 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57tj\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj\") pod \"e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:32.962683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.962643 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:32.969777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:32.969716 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:33.080642 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:33:33.080611 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d14800_0a21_4d34_91c6_ddf985173178.slice/crio-21bf4c01c87ce7164d0d001945fabc8f65c4614f470e554f8453dd9612f1a95c WatchSource:0}: Error finding container 21bf4c01c87ce7164d0d001945fabc8f65c4614f470e554f8453dd9612f1a95c: Status 404 returned error can't find the container with id 21bf4c01c87ce7164d0d001945fabc8f65c4614f470e554f8453dd9612f1a95c Nov 02 17:33:33.081307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.081283 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf"] Nov 02 17:33:33.083548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.083503 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.083636 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.083608 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.083676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.083658 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.173771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.173661 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:33:33.175209 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.175182 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:33:33.338761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.338710 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf"] Nov 02 17:33:33.342021 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:33:33.341979 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89295e9b_c1c4_47a7_9918_1a61b7962c5d.slice/crio-150f1fbbb75c871326b4d1e6acffad4acc4fdbabde3860155b2e9b6099a7b7ee WatchSource:0}: Error finding container 150f1fbbb75c871326b4d1e6acffad4acc4fdbabde3860155b2e9b6099a7b7ee: Status 404 returned error can't find the container with id 150f1fbbb75c871326b4d1e6acffad4acc4fdbabde3860155b2e9b6099a7b7ee Nov 02 17:33:33.344843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.344801 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.344927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.344893 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.344977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.344940 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.390434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.390205 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4"] Nov 02 17:33:33.393652 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:33:33.393618 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633ae065_02b0_427f_b469_c60ab1700d84.slice/crio-3de8d535eb9fb09ba8ae3139302554c6afe7bc9210bf80eb2e6c723373589451 WatchSource:0}: Error finding container 3de8d535eb9fb09ba8ae3139302554c6afe7bc9210bf80eb2e6c723373589451: Status 404 returned error can't find the container with id 3de8d535eb9fb09ba8ae3139302554c6afe7bc9210bf80eb2e6c723373589451 Nov 02 17:33:33.396935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.396880 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.397070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.396979 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.397070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.397028 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:33.912588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.912545 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" event={"ID":"89295e9b-c1c4-47a7-9918-1a61b7962c5d","Type":"ContainerStarted","Data":"bfff14728bd2dc05c545131075dc52292914583ee4f5273c5df0fd2d2134f347"} Nov 02 17:33:33.912781 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.912642 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" event={"ID":"89295e9b-c1c4-47a7-9918-1a61b7962c5d","Type":"ContainerStarted","Data":"150f1fbbb75c871326b4d1e6acffad4acc4fdbabde3860155b2e9b6099a7b7ee"} Nov 02 17:33:33.914549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.914521 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" event={"ID":"76d14800-0a21-4d34-91c6-ddf985173178","Type":"ContainerStarted","Data":"f99d4253afd412e3e17f9861f3966b642ab370a4b6dcb044b6903a07fff5c993"} Nov 02 17:33:33.914712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.914572 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" event={"ID":"76d14800-0a21-4d34-91c6-ddf985173178","Type":"ContainerStarted","Data":"21bf4c01c87ce7164d0d001945fabc8f65c4614f470e554f8453dd9612f1a95c"} Nov 02 17:33:33.916216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.916189 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" event={"ID":"633ae065-02b0-427f-b469-c60ab1700d84","Type":"ContainerStarted","Data":"6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c"} Nov 02 17:33:33.916348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.916233 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" event={"ID":"633ae065-02b0-427f-b469-c60ab1700d84","Type":"ContainerStarted","Data":"3de8d535eb9fb09ba8ae3139302554c6afe7bc9210bf80eb2e6c723373589451"} Nov 02 17:33:33.965346 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.965308 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:33.970400 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:33.970367 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:34.034811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.034720 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podStartSLOduration=2.03469993 podStartE2EDuration="2.03469993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:34.033079295 +0000 UTC m=+601.106606496" watchObservedRunningTime="2026-11-02 17:33:34.03469993 +0000 UTC m=+601.108227135" Nov 02 17:33:34.159325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.159249 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podStartSLOduration=3.159230868 podStartE2EDuration="3.159230868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:34.157965905 +0000 UTC m=+601.231493120" watchObservedRunningTime="2026-11-02 17:33:34.159230868 +0000 UTC m=+601.232758072" Nov 02 17:33:34.289623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.277819 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podStartSLOduration=2.2777966259999998 podStartE2EDuration="2.277796626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:34.275611662 +0000 UTC m=+601.349138866" watchObservedRunningTime="2026-11-02 17:33:34.277796626 +0000 UTC m=+601.351323832" Nov 02 17:33:34.448425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.448388 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7"] Nov 02 17:33:34.451524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.451499 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465464 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465518 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465610 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465652 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465678 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxvv\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465783 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465834 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465819 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465922 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465911 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.465959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.465933 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.506969 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.506931 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3eb29e6e-openshift-default-dockercfg-wlj2s\"" Nov 02 17:33:34.520699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.520033 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7"] Nov 02 17:33:34.567200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567099 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567164 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567197 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567231 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567267 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxvv\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567312 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567347 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567420 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567451 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567598 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567631 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567878 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.567923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.567899 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.568083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.568059 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.569991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.569968 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.570133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.570094 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.629522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.629496 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.629683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.629665 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxvv\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv\") pod \"e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.647408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.647375 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:34.652445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.652425 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:34.763821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.763767 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:34.922290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.922257 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:34.922820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.922795 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:33:34.969572 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.969538 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:34.975511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:34.975480 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:35.158393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.158361 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7"] Nov 02 17:33:35.159142 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:33:35.159115 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629c7574_eba1_421f_aefd_722b69e7f1fd.slice/crio-7588a9224045e15a7ddb865c66af3b5618fcaaa23fa60704b523236b3a459986 WatchSource:0}: Error finding container 7588a9224045e15a7ddb865c66af3b5618fcaaa23fa60704b523236b3a459986: Status 404 returned error can't find the container with id 7588a9224045e15a7ddb865c66af3b5618fcaaa23fa60704b523236b3a459986 Nov 02 17:33:35.161635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.161589 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:35.162077 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.161692 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:35.162077 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.161789 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:35.929181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.929147 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" event={"ID":"629c7574-eba1-421f-aefd-722b69e7f1fd","Type":"ContainerStarted","Data":"80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b"} Nov 02 17:33:35.929371 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.929203 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" event={"ID":"629c7574-eba1-421f-aefd-722b69e7f1fd","Type":"ContainerStarted","Data":"7588a9224045e15a7ddb865c66af3b5618fcaaa23fa60704b523236b3a459986"} Nov 02 17:33:35.929921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.929600 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:35.929921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.929654 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:35.930670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.930648 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:33:35.930797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:35.930788 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:33:35.947955 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:33:35.947927 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:33:36.050836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:36.050778 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podStartSLOduration=2.05075831 podStartE2EDuration="2.05075831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:36.04883579 +0000 UTC m=+603.122362994" watchObservedRunningTime="2026-11-02 17:33:36.05075831 +0000 UTC m=+603.124285512" Nov 02 17:33:36.764447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:36.764409 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:36.769866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:36.769842 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:36.934235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:36.934210 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:36.935255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:36.935234 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:33:48.070216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:48.070182 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:33:50.845833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.845797 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k"] Nov 02 17:33:50.848626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.848603 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:50.963571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.963535 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3eb29e6e\"" Nov 02 17:33:50.963780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.963641 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3eb29e6e-dockercfg-nkcf4\"" Nov 02 17:33:50.965577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.965552 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k"] Nov 02 17:33:50.972549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.972519 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvgp\" (UniqueName: \"kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp\") pod \"payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:50.972678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:50.972580 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume\") pod \"payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:51.074271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.074231 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvgp\" (UniqueName: \"kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp\") pod \"payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:51.074463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.074320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume\") pod \"payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:51.075063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.075038 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume\") pod \"payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:51.132394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.132321 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvgp\" (UniqueName: \"kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp\") pod \"payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:51.161605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.161574 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:51.554941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.554869 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k"] Nov 02 17:33:51.557333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:33:51.557300 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf63ed77_e682_4121_a13e_91217bff04ef.slice/crio-edcdf4ef7f984f5c1a29a5cbc56e9e25533dc44a97f26e9f380a1c4154e78385 WatchSource:0}: Error finding container edcdf4ef7f984f5c1a29a5cbc56e9e25533dc44a97f26e9f380a1c4154e78385: Status 404 returned error can't find the container with id edcdf4ef7f984f5c1a29a5cbc56e9e25533dc44a97f26e9f380a1c4154e78385 Nov 02 17:33:51.559561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:51.559544 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:33:52.053217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:52.053185 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" event={"ID":"cf63ed77-e682-4121-a13e-91217bff04ef","Type":"ContainerStarted","Data":"edcdf4ef7f984f5c1a29a5cbc56e9e25533dc44a97f26e9f380a1c4154e78385"} Nov 02 17:33:55.080977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:55.080943 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" event={"ID":"cf63ed77-e682-4121-a13e-91217bff04ef","Type":"ContainerStarted","Data":"d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415"} Nov 02 17:33:55.081361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:55.081005 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:33:55.193864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:55.193762 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" podStartSLOduration=2.253243897 podStartE2EDuration="5.193720137s" totalImagesPullingTime="2.94047624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:55.19291279 +0000 UTC m=+622.266439994" watchObservedRunningTime="2026-11-02 17:33:55.193720137 +0000 UTC m=+622.267247339" Nov 02 17:33:59.097527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:59.097488 5647 generic.go:356] "Generic (PLEG): container finished" podID="76d14800-0a21-4d34-91c6-ddf985173178" containerID="f99d4253afd412e3e17f9861f3966b642ab370a4b6dcb044b6903a07fff5c993" exitCode=0 Nov 02 17:33:59.097997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:59.097559 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" event={"ID":"76d14800-0a21-4d34-91c6-ddf985173178","Type":"ContainerDied","Data":"f99d4253afd412e3e17f9861f3966b642ab370a4b6dcb044b6903a07fff5c993"} Nov 02 17:33:59.097997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:59.097941 5647 scope.go:122] "RemoveContainer" containerID="f99d4253afd412e3e17f9861f3966b642ab370a4b6dcb044b6903a07fff5c993" Nov 02 17:33:59.099358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:59.099314 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:59.099475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:59.099406 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:33:59.099475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:33:59.099439 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:00.126760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:00.126265 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" event={"ID":"76d14800-0a21-4d34-91c6-ddf985173178","Type":"ContainerStarted","Data":"39936686ab79cb7760c6634f4eaafabb9f895177cd55b32ac9ace7bd55647455"} Nov 02 17:34:00.648312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:00.648273 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:34:00.648692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:00.648666 5647 patch_prober.go:29] interesting pod/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:34:00.648911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:00.648884 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 17:34:01.647090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:01.647056 5647 patch_prober.go:29] interesting pod/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:34:01.647517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:01.647130 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 17:34:02.650838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:02.650358 5647 patch_prober.go:29] interesting pod/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:34:02.650838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:02.650426 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 17:34:02.687510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:02.687468 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:34:03.653208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:03.653161 5647 patch_prober.go:29] interesting pod/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:34:03.654015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:03.653982 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 17:34:04.651594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:04.651122 5647 patch_prober.go:29] interesting pod/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 02 17:34:04.651594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:04.651198 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 02 17:34:04.930817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:04.930415 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:34:05.651605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:05.651575 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:34:05.652520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:05.652498 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:34:05.930800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:05.930693 5647 patch_prober.go:29] interesting pod/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:34:05.930800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:05.930785 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 17:34:05.954519 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:34:05.954482 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:34:06.084092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.084005 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:34:06.183262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.183171 5647 generic.go:356] "Generic (PLEG): container finished" podID="633ae065-02b0-427f-b469-c60ab1700d84" containerID="6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c" exitCode=0 Nov 02 17:34:06.183262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.183242 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" event={"ID":"633ae065-02b0-427f-b469-c60ab1700d84","Type":"ContainerDied","Data":"6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c"} Nov 02 17:34:06.183794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.183774 5647 scope.go:122] "RemoveContainer" containerID="6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c" Nov 02 17:34:06.185036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.185010 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:06.185111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.185077 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:06.185148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:06.185114 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:07.191666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:07.191632 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" event={"ID":"633ae065-02b0-427f-b469-c60ab1700d84","Type":"ContainerStarted","Data":"e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750"} Nov 02 17:34:07.970486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:07.969975 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:34:07.970486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:07.970223 5647 patch_prober.go:29] interesting pod/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:34:07.970486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:07.970268 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 17:34:08.970194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:08.970158 5647 patch_prober.go:29] interesting pod/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:34:08.970587 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:08.970210 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 17:34:09.970536 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:09.970497 5647 patch_prober.go:29] interesting pod/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:34:09.971005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:09.970564 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 17:34:10.970455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:10.970416 5647 patch_prober.go:29] interesting pod/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:34:10.970632 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:10.970490 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 17:34:11.970677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:11.970637 5647 patch_prober.go:29] interesting pod/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 02 17:34:11.971105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:11.970702 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 02 17:34:12.974872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:12.974843 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:34:12.975290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:12.975173 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:34:12.976054 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:12.976034 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:34:14.272484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:14.272443 5647 generic.go:356] "Generic (PLEG): container finished" podID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerID="80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b" exitCode=0 Nov 02 17:34:14.273055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:14.272514 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" event={"ID":"629c7574-eba1-421f-aefd-722b69e7f1fd","Type":"ContainerDied","Data":"80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b"} Nov 02 17:34:14.273055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:14.272929 5647 scope.go:122] "RemoveContainer" containerID="80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b" Nov 02 17:34:14.274126 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:14.274092 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:14.274248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:14.274179 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:14.274248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:14.274229 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:15.279804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:15.279767 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" event={"ID":"629c7574-eba1-421f-aefd-722b69e7f1fd","Type":"ContainerStarted","Data":"94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2"} Nov 02 17:34:15.764521 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:15.764475 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:34:15.764777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:15.764705 5647 patch_prober.go:29] interesting pod/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:34:15.764869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:15.764795 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:34:16.764480 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:16.764439 5647 patch_prober.go:29] interesting pod/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:34:16.764916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:16.764505 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:34:17.764257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:17.764211 5647 patch_prober.go:29] interesting pod/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:34:17.764604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:17.764564 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:34:17.882362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:17.882314 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp"] Nov 02 17:34:17.887319 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:17.887291 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030386 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030450 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030489 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030529 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030622 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030661 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030761 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrjl\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030843 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.030897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.030874 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.065315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.065229 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp"] Nov 02 17:34:18.132384 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132345 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132394 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132434 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132468 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132550 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132587 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132656 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrjl\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132715 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.132860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132787 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.133062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132896 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.133062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132932 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.133062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.132979 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.133257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.133236 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.133597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.133575 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.135172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.135132 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.135941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.135919 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.193855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.193821 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.200542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.200511 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrjl\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.220246 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.219961 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:18.252949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.247876 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:34:18.730764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.719003 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp"] Nov 02 17:34:18.737102 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:34:18.737067 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c317f1_5ab7_4900_a9b3_abf6af356605.slice/crio-e64a6559eb6abb99774285ffb4fb5a94f2fcd7868f800b94d0b6c5771223773a WatchSource:0}: Error finding container e64a6559eb6abb99774285ffb4fb5a94f2fcd7868f800b94d0b6c5771223773a: Status 404 returned error can't find the container with id e64a6559eb6abb99774285ffb4fb5a94f2fcd7868f800b94d0b6c5771223773a Nov 02 17:34:18.739984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.739944 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:18.740100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.740038 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:18.740100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.740086 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:18.765089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.765056 5647 patch_prober.go:29] interesting pod/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:34:18.765415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:18.765123 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:34:19.315766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:19.310375 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" event={"ID":"f4c317f1-5ab7-4900-a9b3-abf6af356605","Type":"ContainerStarted","Data":"c2cd177a92fb1569155a8d8183415f042733d134eb4d4731e8e975199b9888a9"} Nov 02 17:34:19.315766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:19.310451 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" event={"ID":"f4c317f1-5ab7-4900-a9b3-abf6af356605","Type":"ContainerStarted","Data":"e64a6559eb6abb99774285ffb4fb5a94f2fcd7868f800b94d0b6c5771223773a"} Nov 02 17:34:19.446100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:19.446026 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podStartSLOduration=2.446006084 podStartE2EDuration="2.446006084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:19.44393797 +0000 UTC m=+646.517465174" watchObservedRunningTime="2026-11-02 17:34:19.446006084 +0000 UTC m=+646.519533288" Nov 02 17:34:19.764608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:19.764229 5647 patch_prober.go:29] interesting pod/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 02 17:34:19.764608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:19.764288 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 02 17:34:20.225350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.224434 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:34:20.225350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.224500 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 17:34:20.225350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.224688 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:20.768727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.768693 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:34:20.770936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.768998 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:34:20.770936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.769812 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:34:20.931272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.931197 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:34:20.931272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:20.931255 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 17:34:21.224181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.224140 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:34:21.224510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.224467 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 17:34:21.347510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.347475 5647 generic.go:356] "Generic (PLEG): container finished" podID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerID="bfff14728bd2dc05c545131075dc52292914583ee4f5273c5df0fd2d2134f347" exitCode=0 Nov 02 17:34:21.347956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.347550 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" event={"ID":"89295e9b-c1c4-47a7-9918-1a61b7962c5d","Type":"ContainerDied","Data":"bfff14728bd2dc05c545131075dc52292914583ee4f5273c5df0fd2d2134f347"} Nov 02 17:34:21.348183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.348167 5647 scope.go:122] "RemoveContainer" containerID="bfff14728bd2dc05c545131075dc52292914583ee4f5273c5df0fd2d2134f347" Nov 02 17:34:21.349295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.349267 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:21.349367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.349336 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:21.349407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:21.349368 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:22.221226 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.221181 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:34:22.221524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.221492 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 17:34:22.357839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.357798 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" event={"ID":"89295e9b-c1c4-47a7-9918-1a61b7962c5d","Type":"ContainerStarted","Data":"4a364e16911fa1d37189dcbe7d91865da069a03f677eb7ca8aca4df42d1037ee"} Nov 02 17:34:22.776576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.776535 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2"] Nov 02 17:34:22.780537 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.780520 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:22.843726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.843694 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2"] Nov 02 17:34:22.844260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.844237 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:34:22.844778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.844485 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:34:22.959527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.959477 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:22.959527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.959533 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmw8\" (UniqueName: \"kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:22.959841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.959667 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:22.963206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.963177 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:34:22.963484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.963456 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:34:22.963596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:22.963505 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 17:34:23.060879 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.060772 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.061039 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.060902 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.061039 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.060929 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmw8\" (UniqueName: \"kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.061658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.061625 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.063810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.063781 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.145762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.145478 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmw8\" (UniqueName: \"kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8\") pod \"maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.222608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.222149 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:34:23.222608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.222212 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 17:34:23.393307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.393270 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:23.832817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.831901 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2"] Nov 02 17:34:23.963116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.963085 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:34:23.963281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:23.963136 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 17:34:24.241759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:24.236129 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Nov 02 17:34:24.241759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:24.236191 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Nov 02 17:34:24.369764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:24.369040 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" event={"ID":"11f19a28-8379-497d-867f-7ace902533a3","Type":"ContainerStarted","Data":"a458d4d041b3c5550ee4a9a2a4903f8439b617d9212f81c3a21db90b170ff050"} Nov 02 17:34:24.963264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:24.963228 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:34:24.963665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:24.963284 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 17:34:25.228162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:25.228079 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:25.228436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:25.228400 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:25.229507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:25.229490 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:34:25.963127 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:25.963088 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:34:25.963316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:25.963146 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 17:34:26.963297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:26.963258 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 02 17:34:26.963479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:26.963329 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 02 17:34:27.432047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.431425 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp"] Nov 02 17:34:27.436790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.436764 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.497705 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.497623 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp"] Nov 02 17:34:27.533291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.533194 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.533467 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.533317 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjxf\" (UniqueName: \"kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.533467 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.533455 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.634813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.634762 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.634966 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.634881 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.634966 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.634944 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjxf\" (UniqueName: \"kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.635722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.635694 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.638444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.638418 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.694030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.693949 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjxf\" (UniqueName: \"kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf\") pod \"maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.754985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.754948 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:27.968214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.968133 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:34:27.968883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.968833 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:34:27.969866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:27.969841 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:34:28.145557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:28.145529 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp"] Nov 02 17:34:28.403060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:28.403021 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" event={"ID":"c71b2476-ac30-411a-a006-5877f341747f","Type":"ContainerStarted","Data":"fdec34b2cfb7bf8ed6895afe6a011631fac7a04226ad775cad5a553eb9e237b5"} Nov 02 17:34:29.409696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:29.409658 5647 generic.go:356] "Generic (PLEG): container finished" podID="11f19a28-8379-497d-867f-7ace902533a3" containerID="ae92423a132b4598501bd509f1253d611ddf1b020e6d50fcc134e756a40bb109" exitCode=0 Nov 02 17:34:29.410191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:29.409727 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" event={"ID":"11f19a28-8379-497d-867f-7ace902533a3","Type":"ContainerDied","Data":"ae92423a132b4598501bd509f1253d611ddf1b020e6d50fcc134e756a40bb109"} Nov 02 17:34:30.568423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.568397 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:30.689933 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.689852 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca\") pod \"11f19a28-8379-497d-867f-7ace902533a3\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " Nov 02 17:34:30.690110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.689933 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8\" (UniqueName: \"kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8\") pod \"11f19a28-8379-497d-867f-7ace902533a3\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " Nov 02 17:34:30.690110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.690067 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca\") pod \"11f19a28-8379-497d-867f-7ace902533a3\" (UID: \"11f19a28-8379-497d-867f-7ace902533a3\") " Nov 02 17:34:30.690501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.690471 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca" pod "11f19a28-8379-497d-867f-7ace902533a3" (UID: "11f19a28-8379-497d-867f-7ace902533a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:30.692333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.692309 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8" pod "11f19a28-8379-497d-867f-7ace902533a3" (UID: "11f19a28-8379-497d-867f-7ace902533a3"). InnerVolumeSpecName "kube-api-access-mwmw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:30.692456 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.692440 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca" pod "11f19a28-8379-497d-867f-7ace902533a3" (UID: "11f19a28-8379-497d-867f-7ace902533a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:30.791516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.791474 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f19a28-8379-497d-867f-7ace902533a3-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:30.791516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.791509 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmw8\" (UniqueName: \"kubernetes.io/projected/11f19a28-8379-497d-867f-7ace902533a3-kube-api-access-mwmw8\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:30.791725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:30.791525 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f19a28-8379-497d-867f-7ace902533a3-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:31.423171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:31.423139 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" Nov 02 17:34:31.423171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:31.423159 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2" event={"ID":"11f19a28-8379-497d-867f-7ace902533a3","Type":"ContainerDied","Data":"a458d4d041b3c5550ee4a9a2a4903f8439b617d9212f81c3a21db90b170ff050"} Nov 02 17:34:31.423392 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:31.423213 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a458d4d041b3c5550ee4a9a2a4903f8439b617d9212f81c3a21db90b170ff050" Nov 02 17:34:31.424887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:31.424859 5647 generic.go:356] "Generic (PLEG): container finished" podID="c71b2476-ac30-411a-a006-5877f341747f" containerID="260545a83f15b3e6806ab9b26f1d6dc3c90c8b6df2d5aae35c9cb2f85a879354" exitCode=0 Nov 02 17:34:31.425063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:31.424901 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" event={"ID":"c71b2476-ac30-411a-a006-5877f341747f","Type":"ContainerDied","Data":"260545a83f15b3e6806ab9b26f1d6dc3c90c8b6df2d5aae35c9cb2f85a879354"} Nov 02 17:34:32.568102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.568074 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:32.711467 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.711386 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca\") pod \"c71b2476-ac30-411a-a006-5877f341747f\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " Nov 02 17:34:32.711467 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.711443 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca\") pod \"c71b2476-ac30-411a-a006-5877f341747f\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " Nov 02 17:34:32.711751 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.711498 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf\" (UniqueName: \"kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf\") pod \"c71b2476-ac30-411a-a006-5877f341747f\" (UID: \"c71b2476-ac30-411a-a006-5877f341747f\") " Nov 02 17:34:32.711820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.711794 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca" pod "c71b2476-ac30-411a-a006-5877f341747f" (UID: "c71b2476-ac30-411a-a006-5877f341747f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:32.713769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.713720 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca" pod "c71b2476-ac30-411a-a006-5877f341747f" (UID: "c71b2476-ac30-411a-a006-5877f341747f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:32.713932 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.713894 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf" pod "c71b2476-ac30-411a-a006-5877f341747f" (UID: "c71b2476-ac30-411a-a006-5877f341747f"). InnerVolumeSpecName "kube-api-access-8gjxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:32.812805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.812771 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjxf\" (UniqueName: \"kubernetes.io/projected/c71b2476-ac30-411a-a006-5877f341747f-kube-api-access-8gjxf\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:32.812805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.812799 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c71b2476-ac30-411a-a006-5877f341747f-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:32.812805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:32.812809 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c71b2476-ac30-411a-a006-5877f341747f-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:33.436530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:33.436452 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" Nov 02 17:34:33.436530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:33.436485 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp" event={"ID":"c71b2476-ac30-411a-a006-5877f341747f","Type":"ContainerDied","Data":"fdec34b2cfb7bf8ed6895afe6a011631fac7a04226ad775cad5a553eb9e237b5"} Nov 02 17:34:33.436750 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:33.436531 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdec34b2cfb7bf8ed6895afe6a011631fac7a04226ad775cad5a553eb9e237b5" Nov 02 17:34:35.082337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082304 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv"] Nov 02 17:34:35.082761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082747 5647 cpu_manager.go:414] "containerMap: removing container" podUID="11f19a28-8379-497d-867f-7ace902533a3" containerName="revoke-keys" Nov 02 17:34:35.082802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082761 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f19a28-8379-497d-867f-7ace902533a3" containerName="revoke-keys" Nov 02 17:34:35.082802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082790 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c71b2476-ac30-411a-a006-5877f341747f" containerName="revoke-keys" Nov 02 17:34:35.082802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082796 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71b2476-ac30-411a-a006-5877f341747f" containerName="revoke-keys" Nov 02 17:34:35.082895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082862 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71b2476-ac30-411a-a006-5877f341747f" containerName="revoke-keys" Nov 02 17:34:35.082895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.082874 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f19a28-8379-497d-867f-7ace902533a3" containerName="revoke-keys" Nov 02 17:34:35.105129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.105082 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.143459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.143424 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv"] Nov 02 17:34:35.155771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.155701 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rshv7\" (UniqueName: \"kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.155937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.155873 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.156042 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.156023 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.191716 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.191680 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:34:35.191903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.191807 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:34:35.256771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.256725 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.256938 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.256823 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.256938 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.256885 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rshv7\" (UniqueName: \"kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.257466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.257435 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.259326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.259306 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.315417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.315381 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rshv7\" (UniqueName: \"kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7\") pod \"maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.441589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.441555 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:35.813346 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:35.813320 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv"] Nov 02 17:34:35.815585 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:34:35.815553 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda925f18b_81f9_4b96_ab29_c8020507c2e5.slice/crio-d05c9ac5631574e1e06233487dbcd2496e07646badae491ee4c71321ac9cf5f3 WatchSource:0}: Error finding container d05c9ac5631574e1e06233487dbcd2496e07646badae491ee4c71321ac9cf5f3: Status 404 returned error can't find the container with id d05c9ac5631574e1e06233487dbcd2496e07646badae491ee4c71321ac9cf5f3 Nov 02 17:34:35.946235 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:34:35.946201 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:34:36.452824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:36.452718 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" event={"ID":"a925f18b-81f9-4b96-ab29-c8020507c2e5","Type":"ContainerStarted","Data":"cafab92e2ef7a3a896f15a54c295682ece5398a167d538ea01a45c7880fcf17c"} Nov 02 17:34:36.452824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:36.452790 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" event={"ID":"a925f18b-81f9-4b96-ab29-c8020507c2e5","Type":"ContainerStarted","Data":"d05c9ac5631574e1e06233487dbcd2496e07646badae491ee4c71321ac9cf5f3"} Nov 02 17:34:36.569211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:36.569157 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" podStartSLOduration=1.569142228 podStartE2EDuration="1.569142228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:36.567584245 +0000 UTC m=+663.641111450" watchObservedRunningTime="2026-11-02 17:34:36.569142228 +0000 UTC m=+663.642669470" Nov 02 17:34:37.459070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:37.459034 5647 generic.go:356] "Generic (PLEG): container finished" podID="a925f18b-81f9-4b96-ab29-c8020507c2e5" containerID="cafab92e2ef7a3a896f15a54c295682ece5398a167d538ea01a45c7880fcf17c" exitCode=0 Nov 02 17:34:37.459532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:37.459102 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" event={"ID":"a925f18b-81f9-4b96-ab29-c8020507c2e5","Type":"ContainerDied","Data":"cafab92e2ef7a3a896f15a54c295682ece5398a167d538ea01a45c7880fcf17c"} Nov 02 17:34:38.672172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.672141 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:38.705970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.705932 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7\" (UniqueName: \"kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7\") pod \"a925f18b-81f9-4b96-ab29-c8020507c2e5\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " Nov 02 17:34:38.706195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.706022 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca\") pod \"a925f18b-81f9-4b96-ab29-c8020507c2e5\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " Nov 02 17:34:38.706195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.706115 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca\") pod \"a925f18b-81f9-4b96-ab29-c8020507c2e5\" (UID: \"a925f18b-81f9-4b96-ab29-c8020507c2e5\") " Nov 02 17:34:38.706508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.706441 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca" pod "a925f18b-81f9-4b96-ab29-c8020507c2e5" (UID: "a925f18b-81f9-4b96-ab29-c8020507c2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:38.706658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.706633 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:38.709016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.708965 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7" pod "a925f18b-81f9-4b96-ab29-c8020507c2e5" (UID: "a925f18b-81f9-4b96-ab29-c8020507c2e5"). InnerVolumeSpecName "kube-api-access-rshv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:38.709564 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.709533 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca" pod "a925f18b-81f9-4b96-ab29-c8020507c2e5" (UID: "a925f18b-81f9-4b96-ab29-c8020507c2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:38.808364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.808291 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a925f18b-81f9-4b96-ab29-c8020507c2e5-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:38.808364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:38.808334 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rshv7\" (UniqueName: \"kubernetes.io/projected/a925f18b-81f9-4b96-ab29-c8020507c2e5-kube-api-access-rshv7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:39.483342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:39.483264 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" Nov 02 17:34:39.483342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:39.483302 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv" event={"ID":"a925f18b-81f9-4b96-ab29-c8020507c2e5","Type":"ContainerDied","Data":"d05c9ac5631574e1e06233487dbcd2496e07646badae491ee4c71321ac9cf5f3"} Nov 02 17:34:39.483528 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:39.483365 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d05c9ac5631574e1e06233487dbcd2496e07646badae491ee4c71321ac9cf5f3" Nov 02 17:34:43.474455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.474423 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf"] Nov 02 17:34:43.474947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.474932 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a925f18b-81f9-4b96-ab29-c8020507c2e5" containerName="revoke-keys" Nov 02 17:34:43.474991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.474949 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a925f18b-81f9-4b96-ab29-c8020507c2e5" containerName="revoke-keys" Nov 02 17:34:43.475055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.475045 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a925f18b-81f9-4b96-ab29-c8020507c2e5" containerName="revoke-keys" Nov 02 17:34:43.481133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.481114 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.535159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.535127 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf"] Nov 02 17:34:43.562479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.562448 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.562656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.562551 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8fg\" (UniqueName: \"kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.562715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.562662 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.583624 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.583598 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:34:43.583624 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.583596 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:34:43.664073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.664030 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.664256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.664118 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.664256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.664189 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8fg\" (UniqueName: \"kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.664980 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.664950 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.682197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.678418 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.732663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.732572 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8fg\" (UniqueName: \"kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg\") pod \"maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:43.794655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:43.794623 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:44.170242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:44.170213 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf"] Nov 02 17:34:44.171753 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:34:44.171709 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fb8b92_7e26_4036_a8a2_bc8eb3439ee4.slice/crio-e822cb85f18eddcf04ccbdd3d21cb92f324139c10260e114157a1f162b465b3c WatchSource:0}: Error finding container e822cb85f18eddcf04ccbdd3d21cb92f324139c10260e114157a1f162b465b3c: Status 404 returned error can't find the container with id e822cb85f18eddcf04ccbdd3d21cb92f324139c10260e114157a1f162b465b3c Nov 02 17:34:44.511157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:44.511069 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" event={"ID":"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4","Type":"ContainerStarted","Data":"52c26eb5283de798db901194f463cfcab4f86c78c559f955b6911de695196666"} Nov 02 17:34:44.511157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:44.511126 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" event={"ID":"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4","Type":"ContainerStarted","Data":"e822cb85f18eddcf04ccbdd3d21cb92f324139c10260e114157a1f162b465b3c"} Nov 02 17:34:44.642763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:44.642686 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" podStartSLOduration=1.642672173 podStartE2EDuration="1.642672173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:44.642296719 +0000 UTC m=+671.715823922" watchObservedRunningTime="2026-11-02 17:34:44.642672173 +0000 UTC m=+671.716199375" Nov 02 17:34:45.524919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:45.524819 5647 generic.go:356] "Generic (PLEG): container finished" podID="e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" containerID="52c26eb5283de798db901194f463cfcab4f86c78c559f955b6911de695196666" exitCode=0 Nov 02 17:34:45.524919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:45.524864 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" event={"ID":"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4","Type":"ContainerDied","Data":"52c26eb5283de798db901194f463cfcab4f86c78c559f955b6911de695196666"} Nov 02 17:34:46.231527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.231489 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2"] Nov 02 17:34:46.283666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.283626 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa5b913c-q6bz2"] Nov 02 17:34:46.304199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.304161 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf"] Nov 02 17:34:46.304433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.304400 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" containerID="cri-o://39936686ab79cb7760c6634f4eaafabb9f895177cd55b32ac9ace7bd55647455" gracePeriod=30 Nov 02 17:34:46.667270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.667246 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:46.827130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.827051 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca\") pod \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " Nov 02 17:34:46.827299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.827146 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg\" (UniqueName: \"kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg\") pod \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " Nov 02 17:34:46.827299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.827192 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca\") pod \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\" (UID: \"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4\") " Nov 02 17:34:46.827421 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.827396 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca" pod "e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" (UID: "e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:46.827553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.827535 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:46.829348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.829326 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca" pod "e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" (UID: "e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:46.829462 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.829445 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg" pod "e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" (UID: "e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4"). InnerVolumeSpecName "kube-api-access-sk8fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:46.928551 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.928517 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8fg\" (UniqueName: \"kubernetes.io/projected/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-kube-api-access-sk8fg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:46.928551 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:46.928547 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:47.080437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:47.080358 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f19a28-8379-497d-867f-7ace902533a3" path="/var/lib/kubelet/pods/11f19a28-8379-497d-867f-7ace902533a3/volumes" Nov 02 17:34:47.538780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:47.538746 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" Nov 02 17:34:47.538967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:47.538782 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf" event={"ID":"e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4","Type":"ContainerDied","Data":"e822cb85f18eddcf04ccbdd3d21cb92f324139c10260e114157a1f162b465b3c"} Nov 02 17:34:47.538967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:47.538838 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e822cb85f18eddcf04ccbdd3d21cb92f324139c10260e114157a1f162b465b3c" Nov 02 17:34:48.149347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.149296 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp"] Nov 02 17:34:48.151562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.151533 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cc12ac52-5dxbp"] Nov 02 17:34:48.220204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.220173 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4"] Nov 02 17:34:48.220434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.220409 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" containerID="cri-o://e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750" gracePeriod=30 Nov 02 17:34:48.695279 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.695243 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k"] Nov 02 17:34:48.963225 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.963151 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6"] Nov 02 17:34:48.963603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.963590 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" containerName="revoke-keys" Nov 02 17:34:48.963653 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.963604 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" containerName="revoke-keys" Nov 02 17:34:48.963694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.963673 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" containerName="revoke-keys" Nov 02 17:34:48.967081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:48.967063 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.022500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.022468 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:34:49.022672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.022654 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:34:49.024464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.024437 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6"] Nov 02 17:34:49.076714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.076682 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.076917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.076823 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mlkr\" (UniqueName: \"kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.077186 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.077162 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.080983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.080958 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c71b2476-ac30-411a-a006-5877f341747f" path="/var/lib/kubelet/pods/c71b2476-ac30-411a-a006-5877f341747f/volumes" Nov 02 17:34:49.177906 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.177875 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mlkr\" (UniqueName: \"kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.178334 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.177957 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.178334 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.178000 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.178782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.178690 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.180552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.180530 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.235931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.235855 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mlkr\" (UniqueName: \"kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr\") pod \"maas-api-revoke-keys-e2e-derive-135f915e-p4xl6\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.279761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.279715 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:49.597332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.597296 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw"] Nov 02 17:34:49.602416 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.602397 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.656098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.656064 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6e3175-openshift-default-dockercfg-n7khr\"" Nov 02 17:34:49.663527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.663498 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw"] Nov 02 17:34:49.710523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.710499 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6"] Nov 02 17:34:49.712446 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:34:49.712420 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50e54f3_c06e_4db1_bfc8_5db5287fae6b.slice/crio-03a73ee3e860d498371c1087e282d752ce55cb5396269cfdd23c630dd04f1a7c WatchSource:0}: Error finding container 03a73ee3e860d498371c1087e282d752ce55cb5396269cfdd23c630dd04f1a7c: Status 404 returned error can't find the container with id 03a73ee3e860d498371c1087e282d752ce55cb5396269cfdd23c630dd04f1a7c Nov 02 17:34:49.789539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789499 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.789539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789544 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.789792 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789560 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.789792 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789765 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bh9c\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.789952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789803 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.789952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789845 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.789952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789934 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.790090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.789985 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.790090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.790026 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891026 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891067 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891083 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891140 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bh9c\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891163 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891199 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891238 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891282 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891319 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891519 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891580 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891876 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891723 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891876 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891728 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.891965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.891916 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.893717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.893695 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.893965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.893947 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.959785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.959752 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:49.959949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:49.959920 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bh9c\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:50.217183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.216914 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:50.557305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.557200 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" event={"ID":"b50e54f3-c06e-4db1-bfc8-5db5287fae6b","Type":"ContainerStarted","Data":"42af5ebe4b96c28ecc6548a0e398ee53c8b32cdf0dbe69b24a07ead6b2f624e9"} Nov 02 17:34:50.557305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.557268 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" event={"ID":"b50e54f3-c06e-4db1-bfc8-5db5287fae6b","Type":"ContainerStarted","Data":"03a73ee3e860d498371c1087e282d752ce55cb5396269cfdd23c630dd04f1a7c"} Nov 02 17:34:50.643593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.643570 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw"] Nov 02 17:34:50.654424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.654388 5647 patch_prober.go:29] interesting pod/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:34:50.654571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.654452 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:34:50.672793 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:34:50.672752 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a6c65e_5c29_4325_8cc6_1ecef065dba5.slice/crio-57ccbc899555f6593e416f09050758dc950db24399232bf2ae155e29cc27a904 WatchSource:0}: Error finding container 57ccbc899555f6593e416f09050758dc950db24399232bf2ae155e29cc27a904: Status 404 returned error can't find the container with id 57ccbc899555f6593e416f09050758dc950db24399232bf2ae155e29cc27a904 Nov 02 17:34:50.674316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.674271 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" podStartSLOduration=2.674259067 podStartE2EDuration="2.674259067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:50.673649781 +0000 UTC m=+677.747176984" watchObservedRunningTime="2026-11-02 17:34:50.674259067 +0000 UTC m=+677.747786271" Nov 02 17:34:50.675716 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.675681 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:50.675818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.675775 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:50.675818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.675810 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:34:50.771408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:50.771357 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:34:51.564495 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.564462 5647 generic.go:356] "Generic (PLEG): container finished" podID="b50e54f3-c06e-4db1-bfc8-5db5287fae6b" containerID="42af5ebe4b96c28ecc6548a0e398ee53c8b32cdf0dbe69b24a07ead6b2f624e9" exitCode=0 Nov 02 17:34:51.564949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.564541 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" event={"ID":"b50e54f3-c06e-4db1-bfc8-5db5287fae6b","Type":"ContainerDied","Data":"42af5ebe4b96c28ecc6548a0e398ee53c8b32cdf0dbe69b24a07ead6b2f624e9"} Nov 02 17:34:51.568960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.568896 5647 generic.go:356] "Generic (PLEG): container finished" podID="76d14800-0a21-4d34-91c6-ddf985173178" containerID="39936686ab79cb7760c6634f4eaafabb9f895177cd55b32ac9ace7bd55647455" exitCode=0 Nov 02 17:34:51.569104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.568960 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" event={"ID":"76d14800-0a21-4d34-91c6-ddf985173178","Type":"ContainerDied","Data":"39936686ab79cb7760c6634f4eaafabb9f895177cd55b32ac9ace7bd55647455"} Nov 02 17:34:51.569104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.569015 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" event={"ID":"76d14800-0a21-4d34-91c6-ddf985173178","Type":"ContainerDied","Data":"21bf4c01c87ce7164d0d001945fabc8f65c4614f470e554f8453dd9612f1a95c"} Nov 02 17:34:51.569104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.569039 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21bf4c01c87ce7164d0d001945fabc8f65c4614f470e554f8453dd9612f1a95c" Nov 02 17:34:51.569104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.569056 5647 scope.go:122] "RemoveContainer" containerID="f99d4253afd412e3e17f9861f3966b642ab370a4b6dcb044b6903a07fff5c993" Nov 02 17:34:51.569879 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.569859 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:34:51.571558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.571530 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" event={"ID":"b8a6c65e-5c29-4325-8cc6-1ecef065dba5","Type":"ContainerStarted","Data":"89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b"} Nov 02 17:34:51.571655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.571584 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" event={"ID":"b8a6c65e-5c29-4325-8cc6-1ecef065dba5","Type":"ContainerStarted","Data":"57ccbc899555f6593e416f09050758dc950db24399232bf2ae155e29cc27a904"} Nov 02 17:34:51.616506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616476 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616511 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616557 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616575 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616594 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616615 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616662 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.616793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616773 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.617091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.616821 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token\") pod \"76d14800-0a21-4d34-91c6-ddf985173178\" (UID: \"76d14800-0a21-4d34-91c6-ddf985173178\") " Nov 02 17:34:51.617091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617046 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:51.617091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617051 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:51.617250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617102 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:51.617250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617118 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:51.617250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617140 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:51.617411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617376 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.617411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617402 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.617517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617417 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d14800-0a21-4d34-91c6-ddf985173178-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.617517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617433 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.617517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.617492 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.619559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.619531 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:51.619687 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.619568 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:34:51.619687 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.619597 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "kube-api-access-2jrfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:51.619844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.619802 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy" pod "76d14800-0a21-4d34-91c6-ddf985173178" (UID: "76d14800-0a21-4d34-91c6-ddf985173178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:51.718935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.718834 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d14800-0a21-4d34-91c6-ddf985173178-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.718935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.718861 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jrfx\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-kube-api-access-2jrfx\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.718935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.718871 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d14800-0a21-4d34-91c6-ddf985173178-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.718935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.718880 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d14800-0a21-4d34-91c6-ddf985173178-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:51.800680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:51.800623 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podStartSLOduration=2.800609691 podStartE2EDuration="2.800609691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:51.799118342 +0000 UTC m=+678.872645546" watchObservedRunningTime="2026-11-02 17:34:51.800609691 +0000 UTC m=+678.874136893" Nov 02 17:34:52.217290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.217260 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:52.222475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.222448 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:52.578046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.577963 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf" Nov 02 17:34:52.578502 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.578459 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:52.583946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.583920 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:34:52.730281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.730259 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:52.811509 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.811470 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf"] Nov 02 17:34:52.816149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.816124 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-aa5b913c-openshift-default-754ff5d979-2tmtf"] Nov 02 17:34:52.843590 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.843566 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca\") pod \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " Nov 02 17:34:52.843712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.843704 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr\" (UniqueName: \"kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr\") pod \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " Nov 02 17:34:52.843814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.843797 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca\") pod \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\" (UID: \"b50e54f3-c06e-4db1-bfc8-5db5287fae6b\") " Nov 02 17:34:52.844227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.844199 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca" pod "b50e54f3-c06e-4db1-bfc8-5db5287fae6b" (UID: "b50e54f3-c06e-4db1-bfc8-5db5287fae6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:52.845716 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.845684 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca" pod "b50e54f3-c06e-4db1-bfc8-5db5287fae6b" (UID: "b50e54f3-c06e-4db1-bfc8-5db5287fae6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:34:52.845949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.845932 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr" pod "b50e54f3-c06e-4db1-bfc8-5db5287fae6b" (UID: "b50e54f3-c06e-4db1-bfc8-5db5287fae6b"). InnerVolumeSpecName "kube-api-access-8mlkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:52.945901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.945861 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:52.945901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.945902 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mlkr\" (UniqueName: \"kubernetes.io/projected/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-kube-api-access-8mlkr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:52.946143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:52.945918 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50e54f3-c06e-4db1-bfc8-5db5287fae6b-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.081025 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.080993 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d14800-0a21-4d34-91c6-ddf985173178" path="/var/lib/kubelet/pods/76d14800-0a21-4d34-91c6-ddf985173178/volumes" Nov 02 17:34:53.468659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.468620 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:34:53.551429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551335 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551385 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551489 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551569 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551643 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551682 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551714 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551748 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551791 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551763 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551842 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo\") pod \"633ae065-02b0-427f-b469-c60ab1700d84\" (UID: \"633ae065-02b0-427f-b469-c60ab1700d84\") " Nov 02 17:34:53.551860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551841 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:53.552203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551868 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:53.552203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.551989 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:53.552306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.552251 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.552306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.552271 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.552306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.552288 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633ae065-02b0-427f-b469-c60ab1700d84-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.552306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.552302 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.552479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.552318 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.553963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.553930 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:53.554085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.554057 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:34:53.554479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.554456 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:34:53.554579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.554478 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj" pod "633ae065-02b0-427f-b469-c60ab1700d84" (UID: "633ae065-02b0-427f-b469-c60ab1700d84"). InnerVolumeSpecName "kube-api-access-d57tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:53.584986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.584960 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" Nov 02 17:34:53.585426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.584997 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6" event={"ID":"b50e54f3-c06e-4db1-bfc8-5db5287fae6b","Type":"ContainerDied","Data":"03a73ee3e860d498371c1087e282d752ce55cb5396269cfdd23c630dd04f1a7c"} Nov 02 17:34:53.585426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.585073 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03a73ee3e860d498371c1087e282d752ce55cb5396269cfdd23c630dd04f1a7c" Nov 02 17:34:53.587025 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.586998 5647 generic.go:356] "Generic (PLEG): container finished" podID="633ae065-02b0-427f-b469-c60ab1700d84" containerID="e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750" exitCode=0 Nov 02 17:34:53.587175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.587027 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" Nov 02 17:34:53.587175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.587058 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" event={"ID":"633ae065-02b0-427f-b469-c60ab1700d84","Type":"ContainerDied","Data":"e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750"} Nov 02 17:34:53.587175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.587117 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4" event={"ID":"633ae065-02b0-427f-b469-c60ab1700d84","Type":"ContainerDied","Data":"3de8d535eb9fb09ba8ae3139302554c6afe7bc9210bf80eb2e6c723373589451"} Nov 02 17:34:53.587175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.587150 5647 scope.go:122] "RemoveContainer" containerID="e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750" Nov 02 17:34:53.599524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.599500 5647 scope.go:122] "RemoveContainer" containerID="6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c" Nov 02 17:34:53.608276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.608253 5647 scope.go:122] "RemoveContainer" containerID="e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750" Nov 02 17:34:53.608571 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:34:53.608553 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750\": container with ID starting with e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750 not found: ID does not exist" containerID="e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750" Nov 02 17:34:53.608616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.608580 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750"} err="failed to get container status \"e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750\": rpc error: code = NotFound desc = could not find container \"e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750\": container with ID starting with e2f2dc7be1059e074f8d01bc083d3a329b870ad0995e126d1d9a6d91d9832750 not found: ID does not exist" Nov 02 17:34:53.608616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.608598 5647 scope.go:122] "RemoveContainer" containerID="6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c" Nov 02 17:34:53.608917 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:34:53.608893 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c\": container with ID starting with 6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c not found: ID does not exist" containerID="6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c" Nov 02 17:34:53.608970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.608926 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c"} err="failed to get container status \"6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c\": rpc error: code = NotFound desc = could not find container \"6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c\": container with ID starting with 6875c5a8ac9bb1a2e4fca8ed4b4ea48821bfef066c58268e67092218a49aa41c not found: ID does not exist" Nov 02 17:34:53.653882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.653843 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57tj\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-kube-api-access-d57tj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.653882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.653876 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633ae065-02b0-427f-b469-c60ab1700d84-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.653882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.653889 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633ae065-02b0-427f-b469-c60ab1700d84-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.654127 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.653899 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633ae065-02b0-427f-b469-c60ab1700d84-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:53.696517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.696479 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" podUID="cf63ed77-e682-4121-a13e-91217bff04ef" containerName="payload-processing" containerID="cri-o://d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415" gracePeriod=25 Nov 02 17:34:53.812144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.812114 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4"] Nov 02 17:34:53.823281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.823247 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cc12ac52-gw-openshift-default-7d45798b89-7w7h4"] Nov 02 17:34:53.934339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:53.934315 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:34:54.058328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.058290 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume\") pod \"cf63ed77-e682-4121-a13e-91217bff04ef\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " Nov 02 17:34:54.058534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.058408 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp\" (UniqueName: \"kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp\") pod \"cf63ed77-e682-4121-a13e-91217bff04ef\" (UID: \"cf63ed77-e682-4121-a13e-91217bff04ef\") " Nov 02 17:34:54.058994 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.058968 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume" pod "cf63ed77-e682-4121-a13e-91217bff04ef" (UID: "cf63ed77-e682-4121-a13e-91217bff04ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:34:54.061088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.061020 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp" pod "cf63ed77-e682-4121-a13e-91217bff04ef" (UID: "cf63ed77-e682-4121-a13e-91217bff04ef"). InnerVolumeSpecName "kube-api-access-xvvgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:34:54.160615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.160581 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf63ed77-e682-4121-a13e-91217bff04ef-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:54.160615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.160612 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvgp\" (UniqueName: \"kubernetes.io/projected/cf63ed77-e682-4121-a13e-91217bff04ef-kube-api-access-xvvgp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:34:54.597865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.597825 5647 generic.go:356] "Generic (PLEG): container finished" podID="cf63ed77-e682-4121-a13e-91217bff04ef" containerID="d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415" exitCode=0 Nov 02 17:34:54.598771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.597874 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" Nov 02 17:34:54.598771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.597931 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" event={"ID":"cf63ed77-e682-4121-a13e-91217bff04ef","Type":"ContainerDied","Data":"d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415"} Nov 02 17:34:54.598771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.598027 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k" event={"ID":"cf63ed77-e682-4121-a13e-91217bff04ef","Type":"ContainerDied","Data":"edcdf4ef7f984f5c1a29a5cbc56e9e25533dc44a97f26e9f380a1c4154e78385"} Nov 02 17:34:54.598771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.598077 5647 scope.go:122] "RemoveContainer" containerID="d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415" Nov 02 17:34:54.608157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.608140 5647 scope.go:122] "RemoveContainer" containerID="d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415" Nov 02 17:34:54.608447 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:34:54.608427 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415\": container with ID starting with d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415 not found: ID does not exist" containerID="d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415" Nov 02 17:34:54.608505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.608459 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415"} err="failed to get container status \"d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415\": rpc error: code = NotFound desc = could not find container \"d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415\": container with ID starting with d4c4ae18d58c286783131d7c7e1e09c32754eda748337afe961da33c214a8415 not found: ID does not exist" Nov 02 17:34:54.764342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.764313 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k"] Nov 02 17:34:54.768421 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:54.768396 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eb29e6e-79f67cf5b9-mdw6k"] Nov 02 17:34:55.079860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:55.079784 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633ae065-02b0-427f-b469-c60ab1700d84" path="/var/lib/kubelet/pods/633ae065-02b0-427f-b469-c60ab1700d84/volumes" Nov 02 17:34:55.080188 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:34:55.080175 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf63ed77-e682-4121-a13e-91217bff04ef" path="/var/lib/kubelet/pods/cf63ed77-e682-4121-a13e-91217bff04ef/volumes" Nov 02 17:35:03.215846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.215806 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk"] Nov 02 17:35:03.216289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216271 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b50e54f3-c06e-4db1-bfc8-5db5287fae6b" containerName="revoke-keys" Nov 02 17:35:03.216289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216284 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50e54f3-c06e-4db1-bfc8-5db5287fae6b" containerName="revoke-keys" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216297 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cf63ed77-e682-4121-a13e-91217bff04ef" containerName="payload-processing" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216303 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf63ed77-e682-4121-a13e-91217bff04ef" containerName="payload-processing" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216333 5647 cpu_manager.go:414] "containerMap: removing container" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216338 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216345 5647 cpu_manager.go:414] "containerMap: removing container" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216349 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216357 5647 cpu_manager.go:414] "containerMap: removing container" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216361 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216366 5647 cpu_manager.go:414] "containerMap: removing container" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" Nov 02 17:35:03.216368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216370 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" Nov 02 17:35:03.216646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216433 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" Nov 02 17:35:03.216646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216440 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" Nov 02 17:35:03.216646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216445 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="633ae065-02b0-427f-b469-c60ab1700d84" containerName="istio-proxy" Nov 02 17:35:03.216646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216452 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d14800-0a21-4d34-91c6-ddf985173178" containerName="istio-proxy" Nov 02 17:35:03.216646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216459 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50e54f3-c06e-4db1-bfc8-5db5287fae6b" containerName="revoke-keys" Nov 02 17:35:03.216646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.216465 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf63ed77-e682-4121-a13e-91217bff04ef" containerName="payload-processing" Nov 02 17:35:03.221233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.221211 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.329576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.329543 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk"] Nov 02 17:35:03.391937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.391903 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392113 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.391953 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqfrk\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392113 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392049 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392113 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392081 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392199 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392238 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392288 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392388 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.392442 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.392432 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493164 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493072 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493164 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493131 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493172 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493216 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqfrk\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493256 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493286 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493341 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493370 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493405 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493600 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493764 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493774 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.493906 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.493883 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.494433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.494408 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.495695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.495663 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.495919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.495902 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.552061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.552028 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqfrk\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.552364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.552341 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token\") pod \"e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:03.835030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:03.834945 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:04.265672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.265646 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk"] Nov 02 17:35:04.267268 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:35:04.267240 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48c452aa_1ab6_457b_b936_a07338e010fa.slice/crio-459b8be358b9b477b2e4a9c929c3e66dea304bcb317059b4886912cc5510c047 WatchSource:0}: Error finding container 459b8be358b9b477b2e4a9c929c3e66dea304bcb317059b4886912cc5510c047: Status 404 returned error can't find the container with id 459b8be358b9b477b2e4a9c929c3e66dea304bcb317059b4886912cc5510c047 Nov 02 17:35:04.269795 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.269762 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:04.269875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.269831 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:04.269875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.269858 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:04.653896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.653857 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" event={"ID":"48c452aa-1ab6-457b-b936-a07338e010fa","Type":"ContainerStarted","Data":"0559b3cff618d5cc4ad279c05276f07679d01967e3e27969f8e29fe78f12dd9f"} Nov 02 17:35:04.654063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.653927 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" event={"ID":"48c452aa-1ab6-457b-b936-a07338e010fa","Type":"ContainerStarted","Data":"459b8be358b9b477b2e4a9c929c3e66dea304bcb317059b4886912cc5510c047"} Nov 02 17:35:04.812023 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.811963 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" podStartSLOduration=1.811949096 podStartE2EDuration="1.811949096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:04.774573859 +0000 UTC m=+691.848101060" watchObservedRunningTime="2026-11-02 17:35:04.811949096 +0000 UTC m=+691.885476293" Nov 02 17:35:04.838982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:04.838926 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:05.192533 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:05.192497 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv"] Nov 02 17:35:05.195705 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:05.195672 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eb29e6e-tvkhv"] Nov 02 17:35:05.840826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:05.840796 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:05.861798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:05.861754 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7"] Nov 02 17:35:05.862019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:05.861994 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" containerID="cri-o://94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2" gracePeriod=30 Nov 02 17:35:05.976901 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:35:05.976859 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:35:06.665136 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:06.665102 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:06.666189 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:06.666164 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:07.080404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.080303 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a925f18b-81f9-4b96-ab29-c8020507c2e5" path="/var/lib/kubelet/pods/a925f18b-81f9-4b96-ab29-c8020507c2e5/volumes" Nov 02 17:35:07.342698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.342653 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58"] Nov 02 17:35:07.347006 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.346980 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.403134 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.403099 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ceb1ae-openshift-default-dockercfg-pmptc\"" Nov 02 17:35:07.455837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.455804 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58"] Nov 02 17:35:07.461366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461342 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461402 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461464 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461490 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfr46\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461583 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461603 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461745 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461717 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.461780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.461764 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565347 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565408 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565435 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfr46\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565471 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565501 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565533 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565656 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565703 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.565764 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.566226 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.566551 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.566597 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.566869 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.568881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.567749 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.579109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.579041 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.579493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.579464 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.626135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.626039 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.626300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.626282 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfr46\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:07.661618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:07.661159 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:08.108574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.108542 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58"] Nov 02 17:35:08.147382 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:35:08.147348 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e626ae_6f46_4428_a1e4_b97129721f21.slice/crio-538f7ee8f694730fc9a80232719e05708fe6f070267061413d72ae29dd9b3754 WatchSource:0}: Error finding container 538f7ee8f694730fc9a80232719e05708fe6f070267061413d72ae29dd9b3754: Status 404 returned error can't find the container with id 538f7ee8f694730fc9a80232719e05708fe6f070267061413d72ae29dd9b3754 Nov 02 17:35:08.149797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.149763 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:08.149896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.149851 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:08.149953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.149898 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:08.692919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.692880 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" event={"ID":"87e626ae-6f46-4428-a1e4-b97129721f21","Type":"ContainerStarted","Data":"d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11"} Nov 02 17:35:08.693086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.692941 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" event={"ID":"87e626ae-6f46-4428-a1e4-b97129721f21","Type":"ContainerStarted","Data":"538f7ee8f694730fc9a80232719e05708fe6f070267061413d72ae29dd9b3754"} Nov 02 17:35:08.815260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.815208 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podStartSLOduration=1.815191948 podStartE2EDuration="1.815191948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:08.813439652 +0000 UTC m=+695.886966850" watchObservedRunningTime="2026-11-02 17:35:08.815191948 +0000 UTC m=+695.888719150" Nov 02 17:35:08.960892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.960780 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf"] Nov 02 17:35:08.963351 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:08.963328 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-97ba043b-wzvnf"] Nov 02 17:35:09.079886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:09.079851 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4" path="/var/lib/kubelet/pods/e8fb8b92-7e26-4036-a8a2-bc8eb3439ee4/volumes" Nov 02 17:35:09.661676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:09.661635 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:09.667067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:09.667037 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:09.698132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:09.698105 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:09.699245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:09.699226 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:09.773449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:09.773408 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:35:10.471883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:10.471849 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp"] Nov 02 17:35:10.472138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:10.472112 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" containerID="cri-o://c2cd177a92fb1569155a8d8183415f042733d134eb4d4731e8e975199b9888a9" gracePeriod=30 Nov 02 17:35:10.484535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:10.484502 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk"] Nov 02 17:35:10.484787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:10.484727 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" podUID="48c452aa-1ab6-457b-b936-a07338e010fa" containerName="istio-proxy" containerID="cri-o://0559b3cff618d5cc4ad279c05276f07679d01967e3e27969f8e29fe78f12dd9f" gracePeriod=30 Nov 02 17:35:10.521818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:10.521785 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf"] Nov 02 17:35:10.522027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:10.522000 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" containerID="cri-o://4a364e16911fa1d37189dcbe7d91865da069a03f677eb7ca8aca4df42d1037ee" gracePeriod=30 Nov 02 17:35:11.124318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.124290 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:35:11.229313 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229230 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229313 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229273 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229324 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229343 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229380 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229430 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229532 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229553 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:11.229695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229612 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.229695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229669 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data\") pod \"629c7574-eba1-421f-aefd-722b69e7f1fd\" (UID: \"629c7574-eba1-421f-aefd-722b69e7f1fd\") " Nov 02 17:35:11.230033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229986 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:11.230167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.230068 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:11.230167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.229993 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:35:11.230167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.230154 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.230498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.230177 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.230498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.230375 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:11.232000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.231960 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:11.232000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.231990 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:11.232277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.232256 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "kube-api-access-hzxvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:11.232344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.232277 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo" pod "629c7574-eba1-421f-aefd-722b69e7f1fd" (UID: "629c7574-eba1-421f-aefd-722b69e7f1fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:35:11.331481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331444 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629c7574-eba1-421f-aefd-722b69e7f1fd-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.331481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331480 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.331481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331489 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.331720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331498 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.331720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331514 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.331720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331526 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629c7574-eba1-421f-aefd-722b69e7f1fd-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.331720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.331535 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxvv\" (UniqueName: \"kubernetes.io/projected/629c7574-eba1-421f-aefd-722b69e7f1fd-kube-api-access-hzxvv\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:11.710118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.710086 5647 generic.go:356] "Generic (PLEG): container finished" podID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerID="94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2" exitCode=0 Nov 02 17:35:11.710316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.710142 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" Nov 02 17:35:11.710316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.710174 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" event={"ID":"629c7574-eba1-421f-aefd-722b69e7f1fd","Type":"ContainerDied","Data":"94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2"} Nov 02 17:35:11.710316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.710228 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7" event={"ID":"629c7574-eba1-421f-aefd-722b69e7f1fd","Type":"ContainerDied","Data":"7588a9224045e15a7ddb865c66af3b5618fcaaa23fa60704b523236b3a459986"} Nov 02 17:35:11.710316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.710250 5647 scope.go:122] "RemoveContainer" containerID="94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2" Nov 02 17:35:11.726193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.726170 5647 scope.go:122] "RemoveContainer" containerID="80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b" Nov 02 17:35:11.736715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.736688 5647 scope.go:122] "RemoveContainer" containerID="94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2" Nov 02 17:35:11.737060 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:35:11.737033 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2\": container with ID starting with 94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2 not found: ID does not exist" containerID="94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2" Nov 02 17:35:11.737174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.737065 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2"} err="failed to get container status \"94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2\": rpc error: code = NotFound desc = could not find container \"94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2\": container with ID starting with 94da4a4cec8138764326ea2e8ed7826984a098a2b0bf4479ef8d90369d7d1aa2 not found: ID does not exist" Nov 02 17:35:11.737174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.737086 5647 scope.go:122] "RemoveContainer" containerID="80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b" Nov 02 17:35:11.737472 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:35:11.737434 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b\": container with ID starting with 80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b not found: ID does not exist" containerID="80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b" Nov 02 17:35:11.737546 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.737482 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b"} err="failed to get container status \"80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b\": rpc error: code = NotFound desc = could not find container \"80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b\": container with ID starting with 80f84ec40fd0570ed526124a58b1a40d8111674b42c877e2dd52ac724a7c118b not found: ID does not exist" Nov 02 17:35:11.895571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.895532 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7"] Nov 02 17:35:11.897314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897279 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn"] Nov 02 17:35:11.897818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897796 5647 cpu_manager.go:414] "containerMap: removing container" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" Nov 02 17:35:11.897925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897819 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" Nov 02 17:35:11.897925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897843 5647 cpu_manager.go:414] "containerMap: removing container" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" Nov 02 17:35:11.897925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897849 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" Nov 02 17:35:11.898026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897949 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" Nov 02 17:35:11.898026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.897958 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" containerName="istio-proxy" Nov 02 17:35:11.902704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.902681 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937059 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937098 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dlw\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937197 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937251 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937352 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937383 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937409 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937493 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.937541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.937528 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:11.945900 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.945869 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3eb29e6e-openshift-default-6d6f576d67-9hlb7"] Nov 02 17:35:11.959530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.959503 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ad655-openshift-default-dockercfg-jbjjg\"" Nov 02 17:35:11.973247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:11.973163 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn"] Nov 02 17:35:12.041398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.039108 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.041398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.039886 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.041398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.039949 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dlw\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.041398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.040839 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.041398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.040952 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.046023 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.045987 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.046207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.046099 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.046421 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.046397 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.046535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.046518 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.046653 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.046582 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.046951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.046928 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.047036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.047019 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.047084 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.047055 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.047675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.047651 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.050727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.050702 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.066610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.066583 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.102250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.102219 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dlw\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.108118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.108081 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.219200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.219168 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:12.653270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.653242 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn"] Nov 02 17:35:12.655121 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:35:12.655090 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040bc6b3_7fd7_4545_86a9_439ef63e92b0.slice/crio-e7630018caa1ab35a1b2f820e21b1f82f7ea1f1292b5157eb2f1a1bf3da81f3c WatchSource:0}: Error finding container e7630018caa1ab35a1b2f820e21b1f82f7ea1f1292b5157eb2f1a1bf3da81f3c: Status 404 returned error can't find the container with id e7630018caa1ab35a1b2f820e21b1f82f7ea1f1292b5157eb2f1a1bf3da81f3c Nov 02 17:35:12.657424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.657391 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:12.657493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.657458 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:12.657493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.657485 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:12.718635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.718608 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" event={"ID":"040bc6b3-7fd7-4545-86a9-439ef63e92b0","Type":"ContainerStarted","Data":"e7630018caa1ab35a1b2f820e21b1f82f7ea1f1292b5157eb2f1a1bf3da81f3c"} Nov 02 17:35:12.839408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.839353 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podStartSLOduration=1.839338976 podStartE2EDuration="1.839338976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:12.838091147 +0000 UTC m=+699.911618363" watchObservedRunningTime="2026-11-02 17:35:12.839338976 +0000 UTC m=+699.912866178" Nov 02 17:35:12.971794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.971695 5647 patch_prober.go:29] interesting pod/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:35:12.971794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:12.971781 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:35:13.081801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:13.081770 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629c7574-eba1-421f-aefd-722b69e7f1fd" path="/var/lib/kubelet/pods/629c7574-eba1-421f-aefd-722b69e7f1fd/volumes" Nov 02 17:35:13.220253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:13.220221 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:13.225862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:13.225790 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:13.726919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:13.726881 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" event={"ID":"040bc6b3-7fd7-4545-86a9-439ef63e92b0","Type":"ContainerStarted","Data":"19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3"} Nov 02 17:35:13.727112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:13.727096 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:13.728081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:13.728060 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:35:15.741175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.741096 5647 generic.go:356] "Generic (PLEG): container finished" podID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerID="4a364e16911fa1d37189dcbe7d91865da069a03f677eb7ca8aca4df42d1037ee" exitCode=0 Nov 02 17:35:15.741622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.741167 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" event={"ID":"89295e9b-c1c4-47a7-9918-1a61b7962c5d","Type":"ContainerDied","Data":"4a364e16911fa1d37189dcbe7d91865da069a03f677eb7ca8aca4df42d1037ee"} Nov 02 17:35:15.741622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.741221 5647 scope.go:122] "RemoveContainer" containerID="bfff14728bd2dc05c545131075dc52292914583ee4f5273c5df0fd2d2134f347" Nov 02 17:35:15.743065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.743041 5647 generic.go:356] "Generic (PLEG): container finished" podID="48c452aa-1ab6-457b-b936-a07338e010fa" containerID="0559b3cff618d5cc4ad279c05276f07679d01967e3e27969f8e29fe78f12dd9f" exitCode=0 Nov 02 17:35:15.743187 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.743111 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" event={"ID":"48c452aa-1ab6-457b-b936-a07338e010fa","Type":"ContainerDied","Data":"0559b3cff618d5cc4ad279c05276f07679d01967e3e27969f8e29fe78f12dd9f"} Nov 02 17:35:15.749067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.748135 5647 generic.go:356] "Generic (PLEG): container finished" podID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerID="c2cd177a92fb1569155a8d8183415f042733d134eb4d4731e8e975199b9888a9" exitCode=0 Nov 02 17:35:15.749067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.748891 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" event={"ID":"f4c317f1-5ab7-4900-a9b3-abf6af356605","Type":"ContainerDied","Data":"c2cd177a92fb1569155a8d8183415f042733d134eb4d4731e8e975199b9888a9"} Nov 02 17:35:15.804418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.804373 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:15.852777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.848764 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:35:15.856941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.856910 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:35:15.891339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891301 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.891520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891384 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.891520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891418 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.891520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891446 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.891520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891480 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.891520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891515 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.891803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891559 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.891803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891590 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.891803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891613 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.891803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891664 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.891803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891724 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.891803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891772 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.892132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891839 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.892132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891876 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.892132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891908 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.892132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891936 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.892132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891996 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo\") pod \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\" (UID: \"89295e9b-c1c4-47a7-9918-1a61b7962c5d\") " Nov 02 17:35:15.892132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892054 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.892441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892136 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data\") pod \"48c452aa-1ab6-457b-b936-a07338e010fa\" (UID: \"48c452aa-1ab6-457b-b936-a07338e010fa\") " Nov 02 17:35:15.892441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891762 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.892441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.891790 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.892441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892023 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.892441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892127 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:35:15.892715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892352 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.892715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892368 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.892715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892624 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892891 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892911 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48c452aa-1ab6-457b-b936-a07338e010fa-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892925 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892937 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892935 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892952 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892965 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892980 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.892955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.892994 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.893541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.893169 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.894586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.894501 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.894725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.894596 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:35:15.894919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.894893 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.894998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.894940 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:15.895326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.895303 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:35:15.895429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.895410 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "kube-api-access-c5s4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:15.895604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.895578 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token" pod "89295e9b-c1c4-47a7-9918-1a61b7962c5d" (UID: "89295e9b-c1c4-47a7-9918-1a61b7962c5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:15.895715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.895679 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk" pod "48c452aa-1ab6-457b-b936-a07338e010fa" (UID: "48c452aa-1ab6-457b-b936-a07338e010fa"). InnerVolumeSpecName "kube-api-access-mqfrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:15.994305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994217 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994260 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994321 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994378 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994510 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994557 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.994689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994567 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994644 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.994926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994677 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994763 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.994926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994826 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.994926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994835 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:35:15.994926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.994830 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket\") pod \"f4c317f1-5ab7-4900-a9b3-abf6af356605\" (UID: \"f4c317f1-5ab7-4900-a9b3-abf6af356605\") " Nov 02 17:35:15.995199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995044 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:15.995596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995516 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.995596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995545 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5s4k\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-kube-api-access-c5s4k\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.995596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995561 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4c317f1-5ab7-4900-a9b3-abf6af356605-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.995596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995575 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.995596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995590 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.995596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995605 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995619 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqfrk\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-kube-api-access-mqfrk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995634 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995647 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48c452aa-1ab6-457b-b936-a07338e010fa-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995662 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995675 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89295e9b-c1c4-47a7-9918-1a61b7962c5d-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995689 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995703 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995717 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48c452aa-1ab6-457b-b936-a07338e010fa-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.995768 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48c452aa-1ab6-457b-b936-a07338e010fa-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:15.996927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.996897 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:35:15.997221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.997200 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:15.997281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.997229 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "kube-api-access-bmrjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:35:15.997281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:15.997271 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy" pod "f4c317f1-5ab7-4900-a9b3-abf6af356605" (UID: "f4c317f1-5ab7-4900-a9b3-abf6af356605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:35:16.096960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.096923 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:16.096960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.096952 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrjl\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-kube-api-access-bmrjl\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:16.096960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.096962 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:16.097201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.096972 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4c317f1-5ab7-4900-a9b3-abf6af356605-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:35:16.768259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.768173 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" Nov 02 17:35:16.768259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.768164 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf" event={"ID":"89295e9b-c1c4-47a7-9918-1a61b7962c5d","Type":"ContainerDied","Data":"150f1fbbb75c871326b4d1e6acffad4acc4fdbabde3860155b2e9b6099a7b7ee"} Nov 02 17:35:16.768874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.768309 5647 scope.go:122] "RemoveContainer" containerID="4a364e16911fa1d37189dcbe7d91865da069a03f677eb7ca8aca4df42d1037ee" Nov 02 17:35:16.772093 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.772063 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" event={"ID":"48c452aa-1ab6-457b-b936-a07338e010fa","Type":"ContainerDied","Data":"459b8be358b9b477b2e4a9c929c3e66dea304bcb317059b4886912cc5510c047"} Nov 02 17:35:16.772318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.772083 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk" Nov 02 17:35:16.773823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.773789 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" event={"ID":"f4c317f1-5ab7-4900-a9b3-abf6af356605","Type":"ContainerDied","Data":"e64a6559eb6abb99774285ffb4fb5a94f2fcd7868f800b94d0b6c5771223773a"} Nov 02 17:35:16.773934 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.773891 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp" Nov 02 17:35:16.784771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.783600 5647 scope.go:122] "RemoveContainer" containerID="0559b3cff618d5cc4ad279c05276f07679d01967e3e27969f8e29fe78f12dd9f" Nov 02 17:35:16.794101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.794075 5647 scope.go:122] "RemoveContainer" containerID="c2cd177a92fb1569155a8d8183415f042733d134eb4d4731e8e975199b9888a9" Nov 02 17:35:16.943492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.943457 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp"] Nov 02 17:35:17.010800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:16.976930 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-t29xp"] Nov 02 17:35:17.080926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:17.080846 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" path="/var/lib/kubelet/pods/f4c317f1-5ab7-4900-a9b3-abf6af356605/volumes" Nov 02 17:35:17.081125 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:17.081112 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk"] Nov 02 17:35:17.084747 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:17.084704 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-9qnfk"] Nov 02 17:35:17.198531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:17.198502 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf"] Nov 02 17:35:17.204130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:17.204103 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-97ba043b-openshift-default-67597d7f8f-xgfzf"] Nov 02 17:35:19.080179 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:19.080145 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48c452aa-1ab6-457b-b936-a07338e010fa" path="/var/lib/kubelet/pods/48c452aa-1ab6-457b-b936-a07338e010fa/volumes" Nov 02 17:35:19.080800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:19.080779 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" path="/var/lib/kubelet/pods/89295e9b-c1c4-47a7-9918-1a61b7962c5d/volumes" Nov 02 17:35:22.743972 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:22.743938 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6"] Nov 02 17:35:22.794304 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:22.794260 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-135f915e-p4xl6"] Nov 02 17:35:23.079914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:23.079825 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50e54f3-c06e-4db1-bfc8-5db5287fae6b" path="/var/lib/kubelet/pods/b50e54f3-c06e-4db1-bfc8-5db5287fae6b/volumes" Nov 02 17:35:23.770138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:23.770091 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:35:24.239866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.239829 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp"] Nov 02 17:35:24.240295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240272 5647 cpu_manager.go:414] "containerMap: removing container" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" Nov 02 17:35:24.240481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240462 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" Nov 02 17:35:24.240628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240607 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" Nov 02 17:35:24.240628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240627 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" Nov 02 17:35:24.240818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240669 5647 cpu_manager.go:414] "containerMap: removing container" podUID="48c452aa-1ab6-457b-b936-a07338e010fa" containerName="istio-proxy" Nov 02 17:35:24.240818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240676 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48c452aa-1ab6-457b-b936-a07338e010fa" containerName="istio-proxy" Nov 02 17:35:24.240887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240824 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="48c452aa-1ab6-457b-b936-a07338e010fa" containerName="istio-proxy" Nov 02 17:35:24.240887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240833 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" Nov 02 17:35:24.240887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240842 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" Nov 02 17:35:24.240887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.240853 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c317f1-5ab7-4900-a9b3-abf6af356605" containerName="istio-proxy" Nov 02 17:35:24.245431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.245413 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.329182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.329145 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp"] Nov 02 17:35:24.368531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.368505 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dc5689-openshift-default-dockercfg-zkmkm\"" Nov 02 17:35:24.392098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392069 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392106 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392220 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392258 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392278 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392480 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392387 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392500 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392520 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.392563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.392539 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.494220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.494088 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.494993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.494968 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497209 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497182 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497707 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497765 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497797 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497879 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497905 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.497999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.497937 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.498690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.498636 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.499533 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.498638 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.499671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.494901 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.499850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.499161 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.499963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.495769 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.519895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.510613 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.528517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.528438 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.577451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.577415 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.583299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.583267 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:24.858352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:24.858315 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:25.364641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.364619 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp"] Nov 02 17:35:25.366194 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:35:25.366164 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ac16ac_2d3e_4811_a96d_94a6750f77ef.slice/crio-5e4070e2c6c4cd832d909ef0f67ce863c4e2b8add1e0a43ad79fa03a6b5f841f WatchSource:0}: Error finding container 5e4070e2c6c4cd832d909ef0f67ce863c4e2b8add1e0a43ad79fa03a6b5f841f: Status 404 returned error can't find the container with id 5e4070e2c6c4cd832d909ef0f67ce863c4e2b8add1e0a43ad79fa03a6b5f841f Nov 02 17:35:25.368489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.368456 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:25.368576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.368520 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:25.368576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.368548 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:25.839952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.839909 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" event={"ID":"f8ac16ac-2d3e-4811-a96d-94a6750f77ef","Type":"ContainerStarted","Data":"09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506"} Nov 02 17:35:25.840165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.840017 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" event={"ID":"f8ac16ac-2d3e-4811-a96d-94a6750f77ef","Type":"ContainerStarted","Data":"5e4070e2c6c4cd832d909ef0f67ce863c4e2b8add1e0a43ad79fa03a6b5f841f"} Nov 02 17:35:25.859188 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.859150 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:25.963362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:25.963296 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podStartSLOduration=1.963278564 podStartE2EDuration="1.963278564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:35:25.962881031 +0000 UTC m=+713.036408248" watchObservedRunningTime="2026-11-02 17:35:25.963278564 +0000 UTC m=+713.036805768" Nov 02 17:35:26.863448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:26.863414 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:27.849913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:27.849881 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:27.850951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:27.850931 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:35:33.564402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.564358 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:35:33.884917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.884877 5647 generic.go:356] "Generic (PLEG): container finished" podID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerID="89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b" exitCode=0 Nov 02 17:35:33.885128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.884943 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" event={"ID":"b8a6c65e-5c29-4325-8cc6-1ecef065dba5","Type":"ContainerDied","Data":"89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b"} Nov 02 17:35:33.885381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.885363 5647 scope.go:122] "RemoveContainer" containerID="89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b" Nov 02 17:35:33.886801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.886763 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:33.886924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.886853 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:33.886924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:33.886904 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:34.892709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:34.892675 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" event={"ID":"b8a6c65e-5c29-4325-8cc6-1ecef065dba5","Type":"ContainerStarted","Data":"d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180"} Nov 02 17:35:35.224821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:35.224264 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:35:35.226925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:35.226888 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:35:35.227079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:35.226959 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:35:35.953424 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:35:35.953384 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:35:36.222822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:36.222214 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:35:36.222822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:36.222277 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:35:37.222807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:37.222165 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:35:37.222807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:37.222236 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:35:37.578656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:37.578561 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:35:38.227173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:38.227135 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:35:38.227652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:38.227330 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:35:39.226764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:39.224181 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Nov 02 17:35:39.226764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:39.224246 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Nov 02 17:35:40.222119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:40.222088 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:35:40.223019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:40.223001 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 17:35:41.162760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:41.161828 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:35:51.032080 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:51.032042 5647 generic.go:356] "Generic (PLEG): container finished" podID="87e626ae-6f46-4428-a1e4-b97129721f21" containerID="d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11" exitCode=0 Nov 02 17:35:51.032649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:51.032122 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" event={"ID":"87e626ae-6f46-4428-a1e4-b97129721f21","Type":"ContainerDied","Data":"d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11"} Nov 02 17:35:51.032649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:51.032613 5647 scope.go:122] "RemoveContainer" containerID="d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11" Nov 02 17:35:51.034005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:51.033975 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:51.034101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:51.034046 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:51.034101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:51.034078 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:35:52.061105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:52.061058 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" event={"ID":"87e626ae-6f46-4428-a1e4-b97129721f21","Type":"ContainerStarted","Data":"0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b"} Nov 02 17:35:52.662513 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:52.662479 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:52.663404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:52.663385 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:35:52.663460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:52.663442 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:35:53.661603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:53.661569 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:35:53.662042 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:53.661633 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:35:54.662529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:54.662167 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:35:54.662529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:54.662232 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:35:54.698814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:54.698784 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:55.661954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:55.661916 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:35:55.662145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:55.661966 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:35:56.142532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:56.142500 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:35:56.662432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:56.662402 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Nov 02 17:35:56.662609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:56.662457 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Nov 02 17:35:57.686520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:57.686481 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:57.690153 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:57.690123 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 17:35:59.609135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:59.608923 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": read tcp 10.134.0.2:48532->10.134.0.52:15021: read: connection reset by peer" start-of-body= Nov 02 17:35:59.609135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:35:59.608987 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": read tcp 10.134.0.2:48532->10.134.0.52:15021: read: connection reset by peer" Nov 02 17:36:00.106446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:00.106410 5647 generic.go:356] "Generic (PLEG): container finished" podID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerID="19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3" exitCode=0 Nov 02 17:36:00.106622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:00.106484 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" event={"ID":"040bc6b3-7fd7-4545-86a9-439ef63e92b0","Type":"ContainerDied","Data":"19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3"} Nov 02 17:36:00.106914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:00.106894 5647 scope.go:122] "RemoveContainer" containerID="19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3" Nov 02 17:36:00.108282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:00.108254 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:36:00.108361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:00.108326 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:36:00.108361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:00.108356 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:36:01.135887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:01.135850 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" event={"ID":"040bc6b3-7fd7-4545-86a9-439ef63e92b0","Type":"ContainerStarted","Data":"f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016"} Nov 02 17:36:01.223219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:01.223176 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:36:01.223644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:01.223556 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:36:01.223856 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:01.223827 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:36:02.226761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:02.224191 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:36:02.226761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:02.224270 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:36:03.227165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:03.227127 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:36:03.227662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:03.227405 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:36:04.224213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:04.224175 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:36:04.224555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:04.224522 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:36:05.228760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:05.223358 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:36:05.228760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:05.223414 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:36:05.953827 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:36:05.953790 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:36:06.223820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:06.221195 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Nov 02 17:36:06.223820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:06.221283 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Nov 02 17:36:07.231174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:07.231144 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:36:07.231561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:07.231440 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:36:07.232333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:07.232313 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:36:13.891915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:13.891870 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:36:13.892394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:13.891934 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:36:15.215874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.215769 5647 generic.go:356] "Generic (PLEG): container finished" podID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerID="09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506" exitCode=0 Nov 02 17:36:15.215874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.215830 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" event={"ID":"f8ac16ac-2d3e-4811-a96d-94a6750f77ef","Type":"ContainerDied","Data":"09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506"} Nov 02 17:36:15.216492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.216236 5647 scope.go:122] "RemoveContainer" containerID="09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506" Nov 02 17:36:15.217866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.217834 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:36:15.217958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.217921 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:36:15.218003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.217967 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:36:15.438388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:15.438339 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:36:16.233945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:16.233913 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" event={"ID":"f8ac16ac-2d3e-4811-a96d-94a6750f77ef","Type":"ContainerStarted","Data":"78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e"} Nov 02 17:36:16.859106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:16.859065 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:36:16.859346 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:16.859320 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:36:16.859427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:16.859382 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:36:17.858835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:17.858801 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:36:17.859244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:17.858855 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:36:18.859122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:18.859087 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:36:18.859519 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:18.859154 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:36:19.859316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:19.859275 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:36:19.859774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:19.859356 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:36:20.859584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:20.859551 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 02 17:36:20.860046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:20.859601 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 02 17:36:21.862533 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:21.862501 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:36:21.863030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:21.862770 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:36:21.863487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:21.863470 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 17:36:28.190311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:28.190276 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:36:35.947091 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:36:35.947050 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:36:44.483576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:36:44.483544 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:37:05.949779 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:37:05.949712 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:37:06.370481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:06.370448 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:37:24.283160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:24.283074 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:37:32.628577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.628539 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m"] Nov 02 17:37:32.629033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.629018 5647 cpu_manager.go:414] "containerMap: removing container" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" Nov 02 17:37:32.629091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.629046 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89295e9b-c1c4-47a7-9918-1a61b7962c5d" containerName="istio-proxy" Nov 02 17:37:32.631449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.631430 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.728916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.728882 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.729097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.728945 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.729097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.728997 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.729097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.729081 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnz6\" (UniqueName: \"kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.729201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.729169 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.729238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.729224 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.738589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.738559 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"kube-root-ca.crt\"" Nov 02 17:37:32.738589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.738574 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"openshift-service-ca.crt\"" Nov 02 17:37:32.738795 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.738613 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"default-dockercfg-kdg69\"" Nov 02 17:37:32.738795 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.738663 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ceb1ae\"/\"premium-w0-ceb1ae-kserve-self-signed-certs\"" Nov 02 17:37:32.740397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.740373 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m"] Nov 02 17:37:32.830306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830269 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830301 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830332 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnz6\" (UniqueName: \"kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830399 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830424 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830484 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830802 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830834 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.830931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.830878 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.832871 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.832853 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.832988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.832971 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.888334 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.888257 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnz6\" (UniqueName: \"kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6\") pod \"premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:32.942329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:32.942292 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:33.333766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:33.333723 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m"] Nov 02 17:37:33.335623 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:37:33.335596 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511048c8_8540_4625_836a_dac96d03f292.slice/crio-7f9ed19921407e9c614aaba7456579abda2e7fa9c83c05917262b1d959450a2e WatchSource:0}: Error finding container 7f9ed19921407e9c614aaba7456579abda2e7fa9c83c05917262b1d959450a2e: Status 404 returned error can't find the container with id 7f9ed19921407e9c614aaba7456579abda2e7fa9c83c05917262b1d959450a2e Nov 02 17:37:33.651579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:33.651539 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" event={"ID":"511048c8-8540-4625-836a-dac96d03f292","Type":"ContainerStarted","Data":"2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118"} Nov 02 17:37:33.652032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:33.651593 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" event={"ID":"511048c8-8540-4625-836a-dac96d03f292","Type":"ContainerStarted","Data":"7f9ed19921407e9c614aaba7456579abda2e7fa9c83c05917262b1d959450a2e"} Nov 02 17:37:33.768918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:33.768860 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podStartSLOduration=1.768844497 podStartE2EDuration="1.768844497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:33.767660468 +0000 UTC m=+840.841187672" watchObservedRunningTime="2026-11-02 17:37:33.768844497 +0000 UTC m=+840.842371699" Nov 02 17:37:33.942957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:33.942880 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:35.138649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.138615 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg"] Nov 02 17:37:35.141160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.141142 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.199095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.199066 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg"] Nov 02 17:37:35.247182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.247150 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:37:35.247360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.247223 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:37:35.254706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.254679 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.254826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.254808 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzzkk\" (UniqueName: \"kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.254875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.254845 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.356037 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.355999 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.356248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.356084 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzzkk\" (UniqueName: \"kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.356248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.356108 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.356806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.356780 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.358562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.358542 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.415487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.415405 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzzkk\" (UniqueName: \"kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk\") pod \"maas-api-revoke-sub-887cc672-2smgg\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.453912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.453877 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:35.823283 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:35.823256 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg"] Nov 02 17:37:35.824981 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:37:35.824952 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c338d6_41aa_4e13_83b7_08c210f2f3b4.slice/crio-b2e29aee2a32cbe80883fea6980f538807a31a870670d48cc76ff04251960001 WatchSource:0}: Error finding container b2e29aee2a32cbe80883fea6980f538807a31a870670d48cc76ff04251960001: Status 404 returned error can't find the container with id b2e29aee2a32cbe80883fea6980f538807a31a870670d48cc76ff04251960001 Nov 02 17:37:35.946497 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:37:35.946459 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:37:36.674618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:36.674584 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" event={"ID":"08c338d6-41aa-4e13-83b7-08c210f2f3b4","Type":"ContainerStarted","Data":"2624d1332e9485c603180dbce2be15d1121294a8f9702e47d29d5ef9db1603c9"} Nov 02 17:37:36.674991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:36.674645 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" event={"ID":"08c338d6-41aa-4e13-83b7-08c210f2f3b4","Type":"ContainerStarted","Data":"b2e29aee2a32cbe80883fea6980f538807a31a870670d48cc76ff04251960001"} Nov 02 17:37:36.795900 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:36.795835 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" podStartSLOduration=1.79581894 podStartE2EDuration="1.79581894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:36.793344358 +0000 UTC m=+843.866871561" watchObservedRunningTime="2026-11-02 17:37:36.79581894 +0000 UTC m=+843.869346143" Nov 02 17:37:37.682166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:37.682132 5647 generic.go:356] "Generic (PLEG): container finished" podID="08c338d6-41aa-4e13-83b7-08c210f2f3b4" containerID="2624d1332e9485c603180dbce2be15d1121294a8f9702e47d29d5ef9db1603c9" exitCode=0 Nov 02 17:37:37.682622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:37.682223 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" event={"ID":"08c338d6-41aa-4e13-83b7-08c210f2f3b4","Type":"ContainerDied","Data":"2624d1332e9485c603180dbce2be15d1121294a8f9702e47d29d5ef9db1603c9"} Nov 02 17:37:38.821788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.821761 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:38.897280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.897249 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk\" (UniqueName: \"kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk\") pod \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " Nov 02 17:37:38.897495 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.897335 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca\") pod \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " Nov 02 17:37:38.897495 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.897420 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca\") pod \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\" (UID: \"08c338d6-41aa-4e13-83b7-08c210f2f3b4\") " Nov 02 17:37:38.897843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.897814 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca" pod "08c338d6-41aa-4e13-83b7-08c210f2f3b4" (UID: "08c338d6-41aa-4e13-83b7-08c210f2f3b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:37:38.899503 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.899478 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca" pod "08c338d6-41aa-4e13-83b7-08c210f2f3b4" (UID: "08c338d6-41aa-4e13-83b7-08c210f2f3b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:38.899606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.899543 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk" pod "08c338d6-41aa-4e13-83b7-08c210f2f3b4" (UID: "08c338d6-41aa-4e13-83b7-08c210f2f3b4"). InnerVolumeSpecName "kube-api-access-hzzkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:38.999535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.999442 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:37:38.999535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.999473 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzzkk\" (UniqueName: \"kubernetes.io/projected/08c338d6-41aa-4e13-83b7-08c210f2f3b4-kube-api-access-hzzkk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:37:38.999535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:38.999486 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c338d6-41aa-4e13-83b7-08c210f2f3b4-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:37:39.694674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:39.694637 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" event={"ID":"08c338d6-41aa-4e13-83b7-08c210f2f3b4","Type":"ContainerDied","Data":"b2e29aee2a32cbe80883fea6980f538807a31a870670d48cc76ff04251960001"} Nov 02 17:37:39.694674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:39.694670 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg" Nov 02 17:37:39.694920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:39.694698 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2e29aee2a32cbe80883fea6980f538807a31a870670d48cc76ff04251960001" Nov 02 17:37:42.943490 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:42.943457 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:42.956320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:42.956290 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:43.726556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:43.726529 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:37:48.125869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:48.125830 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:37:53.066500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.066458 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9"] Nov 02 17:37:53.067127 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.067105 5647 cpu_manager.go:414] "containerMap: removing container" podUID="08c338d6-41aa-4e13-83b7-08c210f2f3b4" containerName="revoke-keys" Nov 02 17:37:53.067214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.067128 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c338d6-41aa-4e13-83b7-08c210f2f3b4" containerName="revoke-keys" Nov 02 17:37:53.067277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.067267 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c338d6-41aa-4e13-83b7-08c210f2f3b4" containerName="revoke-keys" Nov 02 17:37:53.070465 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.070445 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.126471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.126443 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9"] Nov 02 17:37:53.145171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.145134 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.145317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.145186 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.145374 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.145329 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kfs\" (UniqueName: \"kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.175425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.175393 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:37:53.175597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.175440 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:37:53.245976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.245941 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.245976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.245980 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.246231 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.246029 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kfs\" (UniqueName: \"kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.246663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.246639 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.248595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.248573 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.304279 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.304244 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kfs\" (UniqueName: \"kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs\") pod \"maas-api-revoke-sub-b2124691-gzft9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.383397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.383364 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:53.754812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.754786 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9"] Nov 02 17:37:53.756388 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:37:53.756362 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47dd25fe_423b_45da_8929_c42b9c60f1c9.slice/crio-900c63e94510393ba81db39a028a6b3a3ef629eed83476ca34e03419e784d17f WatchSource:0}: Error finding container 900c63e94510393ba81db39a028a6b3a3ef629eed83476ca34e03419e784d17f: Status 404 returned error can't find the container with id 900c63e94510393ba81db39a028a6b3a3ef629eed83476ca34e03419e784d17f Nov 02 17:37:53.787578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:53.787542 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" event={"ID":"47dd25fe-423b-45da-8929-c42b9c60f1c9","Type":"ContainerStarted","Data":"900c63e94510393ba81db39a028a6b3a3ef629eed83476ca34e03419e784d17f"} Nov 02 17:37:54.794122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:54.794090 5647 generic.go:356] "Generic (PLEG): container finished" podID="47dd25fe-423b-45da-8929-c42b9c60f1c9" containerID="b1ef8dc1657d9b3435aad802b8c102525e2e32ea1af7cd8b627c8bf4d1e36127" exitCode=0 Nov 02 17:37:54.794532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:54.794130 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" event={"ID":"47dd25fe-423b-45da-8929-c42b9c60f1c9","Type":"ContainerDied","Data":"b1ef8dc1657d9b3435aad802b8c102525e2e32ea1af7cd8b627c8bf4d1e36127"} Nov 02 17:37:55.947902 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.947872 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:55.983875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.983838 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca\") pod \"47dd25fe-423b-45da-8929-c42b9c60f1c9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " Nov 02 17:37:55.984073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.983884 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs\" (UniqueName: \"kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs\") pod \"47dd25fe-423b-45da-8929-c42b9c60f1c9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " Nov 02 17:37:55.984073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.983930 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca\") pod \"47dd25fe-423b-45da-8929-c42b9c60f1c9\" (UID: \"47dd25fe-423b-45da-8929-c42b9c60f1c9\") " Nov 02 17:37:55.984274 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.984251 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca" pod "47dd25fe-423b-45da-8929-c42b9c60f1c9" (UID: "47dd25fe-423b-45da-8929-c42b9c60f1c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:37:55.984418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.984401 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:37:55.986323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.986290 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca" pod "47dd25fe-423b-45da-8929-c42b9c60f1c9" (UID: "47dd25fe-423b-45da-8929-c42b9c60f1c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:37:55.986449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:55.986422 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs" pod "47dd25fe-423b-45da-8929-c42b9c60f1c9" (UID: "47dd25fe-423b-45da-8929-c42b9c60f1c9"). InnerVolumeSpecName "kube-api-access-j8kfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:37:56.085331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:56.085230 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kfs\" (UniqueName: \"kubernetes.io/projected/47dd25fe-423b-45da-8929-c42b9c60f1c9-kube-api-access-j8kfs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:37:56.085331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:56.085263 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47dd25fe-423b-45da-8929-c42b9c60f1c9-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:37:56.806394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:56.806364 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" Nov 02 17:37:56.806579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:56.806399 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9" event={"ID":"47dd25fe-423b-45da-8929-c42b9c60f1c9","Type":"ContainerDied","Data":"900c63e94510393ba81db39a028a6b3a3ef629eed83476ca34e03419e784d17f"} Nov 02 17:37:56.806579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:37:56.806449 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="900c63e94510393ba81db39a028a6b3a3ef629eed83476ca34e03419e784d17f" Nov 02 17:38:05.586318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:05.586280 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:38:05.948010 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:38:05.947923 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:38:06.383381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:06.383329 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp"] Nov 02 17:38:06.383647 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:06.383607 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" containerID="cri-o://eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82" gracePeriod=30 Nov 02 17:38:11.672077 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.672043 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:38:11.755119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755081 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755174 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755228 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755280 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755318 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755349 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755435 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755488 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755500 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755508 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:11.755542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755531 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo\") pod \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\" (UID: \"e02cefeb-4a1f-41b8-bf36-50ccb45f3333\") " Nov 02 17:38:11.755915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755822 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:38:11.755915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.755900 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:38:11.756298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.756123 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:38:11.756298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.756129 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.756298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.756156 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.756298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.756171 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.756298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.756179 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.757997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.757960 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "kube-api-access-c6s9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:11.758211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.758181 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:38:11.758320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.758253 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:11.758320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.758288 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo" pod "e02cefeb-4a1f-41b8-bf36-50ccb45f3333" (UID: "e02cefeb-4a1f-41b8-bf36-50ccb45f3333"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:38:11.857420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.857386 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.857420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.857418 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.857633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.857430 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.857633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.857444 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.857633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.857458 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6s9m\" (UniqueName: \"kubernetes.io/projected/e02cefeb-4a1f-41b8-bf36-50ccb45f3333-kube-api-access-c6s9m\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:11.918391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.918300 5647 generic.go:356] "Generic (PLEG): container finished" podID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerID="eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82" exitCode=0 Nov 02 17:38:11.918391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.918345 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" Nov 02 17:38:11.918391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.918354 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" event={"ID":"e02cefeb-4a1f-41b8-bf36-50ccb45f3333","Type":"ContainerDied","Data":"eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82"} Nov 02 17:38:11.918678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.918408 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp" event={"ID":"e02cefeb-4a1f-41b8-bf36-50ccb45f3333","Type":"ContainerDied","Data":"f6acd838596b3e96ef1de167ec3ce6ec978e3621cbc917e5c09f611306c1e8d0"} Nov 02 17:38:11.918678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.918433 5647 scope.go:122] "RemoveContainer" containerID="eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82" Nov 02 17:38:11.929862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.929841 5647 scope.go:122] "RemoveContainer" containerID="eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82" Nov 02 17:38:11.930230 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:38:11.930197 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82\": container with ID starting with eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82 not found: ID does not exist" containerID="eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82" Nov 02 17:38:11.930286 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:11.930241 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82"} err="failed to get container status \"eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82\": rpc error: code = NotFound desc = could not find container \"eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82\": container with ID starting with eb844dd27fa251e2662854131a5254adf91b7c4d9b61c5d175dc5c8c57423d82 not found: ID does not exist" Nov 02 17:38:12.085804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:12.085768 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp"] Nov 02 17:38:12.090725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:12.090694 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-wb5mp"] Nov 02 17:38:13.079721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:13.079680 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" path="/var/lib/kubelet/pods/e02cefeb-4a1f-41b8-bf36-50ccb45f3333/volumes" Nov 02 17:38:14.331151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331109 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k"] Nov 02 17:38:14.331774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331751 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" Nov 02 17:38:14.331774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331773 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" Nov 02 17:38:14.331926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331787 5647 cpu_manager.go:414] "containerMap: removing container" podUID="47dd25fe-423b-45da-8929-c42b9c60f1c9" containerName="revoke-keys" Nov 02 17:38:14.331926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331794 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dd25fe-423b-45da-8929-c42b9c60f1c9" containerName="revoke-keys" Nov 02 17:38:14.332036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331931 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e02cefeb-4a1f-41b8-bf36-50ccb45f3333" containerName="istio-proxy" Nov 02 17:38:14.332036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.331990 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dd25fe-423b-45da-8929-c42b9c60f1c9" containerName="revoke-keys" Nov 02 17:38:14.336432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.336413 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.441496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.441460 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:38:14.441496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.441488 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:38:14.443249 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.443227 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k"] Nov 02 17:38:14.484947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.484913 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.485133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.484986 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7nft\" (UniqueName: \"kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.485133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.485059 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.586097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.586070 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.586262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.586144 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7nft\" (UniqueName: \"kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.586262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.586164 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.586821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.586804 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.588619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.588594 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.643970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.643938 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7nft\" (UniqueName: \"kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft\") pod \"maas-api-revoke-sub-3ded7d84-sqh2k\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:14.649645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:14.649618 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:15.021376 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.021351 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k"] Nov 02 17:38:15.024230 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:38:15.024192 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1aef59_f21d_4f43_8ac9_de071d307858.slice/crio-79acf7ed14ac92aeab9ce7e7d2e01260521732783104dd3e61e59a680ea65430 WatchSource:0}: Error finding container 79acf7ed14ac92aeab9ce7e7d2e01260521732783104dd3e61e59a680ea65430: Status 404 returned error can't find the container with id 79acf7ed14ac92aeab9ce7e7d2e01260521732783104dd3e61e59a680ea65430 Nov 02 17:38:15.122987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.122944 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws"] Nov 02 17:38:15.127809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.127786 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.185180 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.185131 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws"] Nov 02 17:38:15.294606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.294517 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tv9v\" (UniqueName: \"kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.294785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.294663 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.294785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.294724 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.395954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.395921 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.396358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.395970 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.396358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.396051 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tv9v\" (UniqueName: \"kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.396654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.396634 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.398390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.398371 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.456699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.456669 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tv9v\" (UniqueName: \"kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v\") pod \"maas-api-revoke-sub-dcc889c6-g2sws\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.740768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.740704 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:15.942748 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.942705 5647 generic.go:356] "Generic (PLEG): container finished" podID="5a1aef59-f21d-4f43-8ac9-de071d307858" containerID="b1283ca937d38853f54ef30547919b76138e8cc8d562891843098c4500b6e674" exitCode=0 Nov 02 17:38:15.942916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.942767 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" event={"ID":"5a1aef59-f21d-4f43-8ac9-de071d307858","Type":"ContainerDied","Data":"b1283ca937d38853f54ef30547919b76138e8cc8d562891843098c4500b6e674"} Nov 02 17:38:15.942916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:15.942857 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" event={"ID":"5a1aef59-f21d-4f43-8ac9-de071d307858","Type":"ContainerStarted","Data":"79acf7ed14ac92aeab9ce7e7d2e01260521732783104dd3e61e59a680ea65430"} Nov 02 17:38:16.116274 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:16.116242 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws"] Nov 02 17:38:16.155147 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:38:16.155119 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bf8309_a7fa_4b92_a59f_cbad489d0bc2.slice/crio-7a3a34a9e3c9b16528b7b2d3969266e8f70bb79ab914e4621038dda5dc8068fc WatchSource:0}: Error finding container 7a3a34a9e3c9b16528b7b2d3969266e8f70bb79ab914e4621038dda5dc8068fc: Status 404 returned error can't find the container with id 7a3a34a9e3c9b16528b7b2d3969266e8f70bb79ab914e4621038dda5dc8068fc Nov 02 17:38:16.950231 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:16.950196 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" event={"ID":"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2","Type":"ContainerStarted","Data":"bf275f33e0290f9645cf50d7f01393a8ce25e1563874ecf53b42a238d90aa68f"} Nov 02 17:38:16.950634 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:16.950247 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" event={"ID":"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2","Type":"ContainerStarted","Data":"7a3a34a9e3c9b16528b7b2d3969266e8f70bb79ab914e4621038dda5dc8068fc"} Nov 02 17:38:17.070047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.069995 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" podStartSLOduration=2.069980593 podStartE2EDuration="2.069980593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:17.068391173 +0000 UTC m=+884.141918377" watchObservedRunningTime="2026-11-02 17:38:17.069980593 +0000 UTC m=+884.143507797" Nov 02 17:38:17.083412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.083389 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:17.231088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.231012 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft\" (UniqueName: \"kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft\") pod \"5a1aef59-f21d-4f43-8ac9-de071d307858\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " Nov 02 17:38:17.231267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.231101 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca\") pod \"5a1aef59-f21d-4f43-8ac9-de071d307858\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " Nov 02 17:38:17.231267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.231125 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca\") pod \"5a1aef59-f21d-4f43-8ac9-de071d307858\" (UID: \"5a1aef59-f21d-4f43-8ac9-de071d307858\") " Nov 02 17:38:17.231557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.231536 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca" pod "5a1aef59-f21d-4f43-8ac9-de071d307858" (UID: "5a1aef59-f21d-4f43-8ac9-de071d307858"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:17.233491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.233460 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft" pod "5a1aef59-f21d-4f43-8ac9-de071d307858" (UID: "5a1aef59-f21d-4f43-8ac9-de071d307858"). InnerVolumeSpecName "kube-api-access-f7nft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:17.233611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.233493 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca" pod "5a1aef59-f21d-4f43-8ac9-de071d307858" (UID: "5a1aef59-f21d-4f43-8ac9-de071d307858"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:17.332427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.332391 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7nft\" (UniqueName: \"kubernetes.io/projected/5a1aef59-f21d-4f43-8ac9-de071d307858-kube-api-access-f7nft\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:17.332427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.332423 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:17.332427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.332433 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1aef59-f21d-4f43-8ac9-de071d307858-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:17.957599 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.957560 5647 generic.go:356] "Generic (PLEG): container finished" podID="a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" containerID="bf275f33e0290f9645cf50d7f01393a8ce25e1563874ecf53b42a238d90aa68f" exitCode=0 Nov 02 17:38:17.958136 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.957641 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" event={"ID":"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2","Type":"ContainerDied","Data":"bf275f33e0290f9645cf50d7f01393a8ce25e1563874ecf53b42a238d90aa68f"} Nov 02 17:38:17.959085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.959063 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" Nov 02 17:38:17.959205 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.959064 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k" event={"ID":"5a1aef59-f21d-4f43-8ac9-de071d307858","Type":"ContainerDied","Data":"79acf7ed14ac92aeab9ce7e7d2e01260521732783104dd3e61e59a680ea65430"} Nov 02 17:38:17.959205 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:17.959180 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79acf7ed14ac92aeab9ce7e7d2e01260521732783104dd3e61e59a680ea65430" Nov 02 17:38:19.119131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.119101 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:19.253354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.253266 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca\") pod \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " Nov 02 17:38:19.253585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.253388 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca\") pod \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " Nov 02 17:38:19.253585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.253433 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v\" (UniqueName: \"kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v\") pod \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\" (UID: \"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2\") " Nov 02 17:38:19.253799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.253770 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca" pod "a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" (UID: "a6bf8309-a7fa-4b92-a59f-cbad489d0bc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:19.256330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.256302 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca" pod "a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" (UID: "a6bf8309-a7fa-4b92-a59f-cbad489d0bc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:19.256418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.256306 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v" pod "a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" (UID: "a6bf8309-a7fa-4b92-a59f-cbad489d0bc2"). InnerVolumeSpecName "kube-api-access-7tv9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:19.354664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.354623 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:19.354664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.354656 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tv9v\" (UniqueName: \"kubernetes.io/projected/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-kube-api-access-7tv9v\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:19.354664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.354669 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:19.971683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.971654 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" Nov 02 17:38:19.971868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.971655 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws" event={"ID":"a6bf8309-a7fa-4b92-a59f-cbad489d0bc2","Type":"ContainerDied","Data":"7a3a34a9e3c9b16528b7b2d3969266e8f70bb79ab914e4621038dda5dc8068fc"} Nov 02 17:38:19.971868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:19.971820 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a3a34a9e3c9b16528b7b2d3969266e8f70bb79ab914e4621038dda5dc8068fc" Nov 02 17:38:22.636672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:22.636636 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:38:30.029439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.029401 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx"] Nov 02 17:38:30.029957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.029901 5647 cpu_manager.go:414] "containerMap: removing container" podUID="5a1aef59-f21d-4f43-8ac9-de071d307858" containerName="revoke-keys" Nov 02 17:38:30.029957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.029915 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1aef59-f21d-4f43-8ac9-de071d307858" containerName="revoke-keys" Nov 02 17:38:30.029957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.029925 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" containerName="revoke-keys" Nov 02 17:38:30.029957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.029930 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" containerName="revoke-keys" Nov 02 17:38:30.030088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.030000 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1aef59-f21d-4f43-8ac9-de071d307858" containerName="revoke-keys" Nov 02 17:38:30.030088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.030007 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" containerName="revoke-keys" Nov 02 17:38:30.034500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.034481 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.074096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.074058 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sltnr\" (UniqueName: \"kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.074288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.074135 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.074333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.074286 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.088159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.088119 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:38:30.090082 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.090055 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx"] Nov 02 17:38:30.128871 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.128841 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:38:30.175208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.175169 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sltnr\" (UniqueName: \"kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.175393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.175253 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.175393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.175312 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.176093 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.176070 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.177935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.177919 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.233875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.233841 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sltnr\" (UniqueName: \"kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr\") pod \"maas-api-revoke-sub-85652222-8g7sx\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.347855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.347822 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:30.724107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:30.724071 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx"] Nov 02 17:38:30.725785 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:38:30.725725 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a17321_eccc_45cc_923e_2f981fb59aa6.slice/crio-a47c85ee0bf2258b0f6fba65e42725801f2eb9e1cb3d1398c57274084f25977a WatchSource:0}: Error finding container a47c85ee0bf2258b0f6fba65e42725801f2eb9e1cb3d1398c57274084f25977a: Status 404 returned error can't find the container with id a47c85ee0bf2258b0f6fba65e42725801f2eb9e1cb3d1398c57274084f25977a Nov 02 17:38:31.036085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:31.036043 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" event={"ID":"c5a17321-eccc-45cc-923e-2f981fb59aa6","Type":"ContainerStarted","Data":"13988390fdc0333d763f7f275c343e75df85273560cd3723bcb796e09228da29"} Nov 02 17:38:31.036534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:31.036103 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" event={"ID":"c5a17321-eccc-45cc-923e-2f981fb59aa6","Type":"ContainerStarted","Data":"a47c85ee0bf2258b0f6fba65e42725801f2eb9e1cb3d1398c57274084f25977a"} Nov 02 17:38:31.154065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:31.154011 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" podStartSLOduration=2.153997383 podStartE2EDuration="2.153997383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:31.15267247 +0000 UTC m=+898.226199674" watchObservedRunningTime="2026-11-02 17:38:31.153997383 +0000 UTC m=+898.227524579" Nov 02 17:38:32.043829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:32.043794 5647 generic.go:356] "Generic (PLEG): container finished" podID="c5a17321-eccc-45cc-923e-2f981fb59aa6" containerID="13988390fdc0333d763f7f275c343e75df85273560cd3723bcb796e09228da29" exitCode=0 Nov 02 17:38:32.044233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:32.043840 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" event={"ID":"c5a17321-eccc-45cc-923e-2f981fb59aa6","Type":"ContainerDied","Data":"13988390fdc0333d763f7f275c343e75df85273560cd3723bcb796e09228da29"} Nov 02 17:38:33.191278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.191250 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:33.224463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.224435 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:38:33.227661 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.227636 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:38:33.305828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.305722 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr\" (UniqueName: \"kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr\") pod \"c5a17321-eccc-45cc-923e-2f981fb59aa6\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " Nov 02 17:38:33.306681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.305875 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca\") pod \"c5a17321-eccc-45cc-923e-2f981fb59aa6\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " Nov 02 17:38:33.306681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.305966 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca\") pod \"c5a17321-eccc-45cc-923e-2f981fb59aa6\" (UID: \"c5a17321-eccc-45cc-923e-2f981fb59aa6\") " Nov 02 17:38:33.306681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.306481 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca" pod "c5a17321-eccc-45cc-923e-2f981fb59aa6" (UID: "c5a17321-eccc-45cc-923e-2f981fb59aa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:33.308607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.308581 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr" pod "c5a17321-eccc-45cc-923e-2f981fb59aa6" (UID: "c5a17321-eccc-45cc-923e-2f981fb59aa6"). InnerVolumeSpecName "kube-api-access-sltnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:33.308715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.308578 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca" pod "c5a17321-eccc-45cc-923e-2f981fb59aa6" (UID: "c5a17321-eccc-45cc-923e-2f981fb59aa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:33.407436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.407389 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sltnr\" (UniqueName: \"kubernetes.io/projected/c5a17321-eccc-45cc-923e-2f981fb59aa6-kube-api-access-sltnr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:33.407436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.407425 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:33.407436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:33.407440 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5a17321-eccc-45cc-923e-2f981fb59aa6-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:34.056473 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:34.056438 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" Nov 02 17:38:34.056652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:34.056464 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx" event={"ID":"c5a17321-eccc-45cc-923e-2f981fb59aa6","Type":"ContainerDied","Data":"a47c85ee0bf2258b0f6fba65e42725801f2eb9e1cb3d1398c57274084f25977a"} Nov 02 17:38:34.056652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:34.056539 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47c85ee0bf2258b0f6fba65e42725801f2eb9e1cb3d1398c57274084f25977a" Nov 02 17:38:35.544813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.544771 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv"] Nov 02 17:38:35.545394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.545373 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c5a17321-eccc-45cc-923e-2f981fb59aa6" containerName="revoke-keys" Nov 02 17:38:35.545487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.545396 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a17321-eccc-45cc-923e-2f981fb59aa6" containerName="revoke-keys" Nov 02 17:38:35.545557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.545543 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a17321-eccc-45cc-923e-2f981fb59aa6" containerName="revoke-keys" Nov 02 17:38:35.549813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.549793 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.603596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.603490 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:38:35.629952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.629919 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.630138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.629975 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9sj\" (UniqueName: \"kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.630138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.630094 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.654259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.654226 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:38:35.656112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.656090 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv"] Nov 02 17:38:35.731476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.731443 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.731654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.731549 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.731654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.731590 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9sj\" (UniqueName: \"kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.732287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.732267 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.734016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.733997 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.790078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.790039 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9sj\" (UniqueName: \"kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj\") pod \"maas-api-revoke-sub-9338cdef-8g7bv\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.864473 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:35.864437 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:35.954210 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:38:35.954167 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:38:36.249962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:36.249941 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv"] Nov 02 17:38:36.251854 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:38:36.251830 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abef49e_37f5_442a_9881_769d93359722.slice/crio-fac320d032e44f1059236d05749845bc4e2fd4333b5d768197f545547b8e036c WatchSource:0}: Error finding container fac320d032e44f1059236d05749845bc4e2fd4333b5d768197f545547b8e036c: Status 404 returned error can't find the container with id fac320d032e44f1059236d05749845bc4e2fd4333b5d768197f545547b8e036c Nov 02 17:38:37.079411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:37.079379 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" event={"ID":"9abef49e-37f5-442a-9881-769d93359722","Type":"ContainerStarted","Data":"e62bb2cefb5bb60b74ca8189581622fa0e9725d96ed21716e85c12cb3881ef45"} Nov 02 17:38:37.079874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:37.079430 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" event={"ID":"9abef49e-37f5-442a-9881-769d93359722","Type":"ContainerStarted","Data":"fac320d032e44f1059236d05749845bc4e2fd4333b5d768197f545547b8e036c"} Nov 02 17:38:37.196629 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:37.196569 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" podStartSLOduration=2.196552214 podStartE2EDuration="2.196552214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:37.195994416 +0000 UTC m=+904.269521620" watchObservedRunningTime="2026-11-02 17:38:37.196552214 +0000 UTC m=+904.270079430" Nov 02 17:38:38.101516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:38.101471 5647 generic.go:356] "Generic (PLEG): container finished" podID="9abef49e-37f5-442a-9881-769d93359722" containerID="e62bb2cefb5bb60b74ca8189581622fa0e9725d96ed21716e85c12cb3881ef45" exitCode=0 Nov 02 17:38:38.102057 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:38.101689 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" event={"ID":"9abef49e-37f5-442a-9881-769d93359722","Type":"ContainerDied","Data":"e62bb2cefb5bb60b74ca8189581622fa0e9725d96ed21716e85c12cb3881ef45"} Nov 02 17:38:39.246689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.246660 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:39.374772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.374658 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj\" (UniqueName: \"kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj\") pod \"9abef49e-37f5-442a-9881-769d93359722\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " Nov 02 17:38:39.375037 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.375012 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca\") pod \"9abef49e-37f5-442a-9881-769d93359722\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " Nov 02 17:38:39.375267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.375247 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca\") pod \"9abef49e-37f5-442a-9881-769d93359722\" (UID: \"9abef49e-37f5-442a-9881-769d93359722\") " Nov 02 17:38:39.376066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.376040 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca" pod "9abef49e-37f5-442a-9881-769d93359722" (UID: "9abef49e-37f5-442a-9881-769d93359722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:39.378235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.378208 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca" pod "9abef49e-37f5-442a-9881-769d93359722" (UID: "9abef49e-37f5-442a-9881-769d93359722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:39.378235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.378230 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj" pod "9abef49e-37f5-442a-9881-769d93359722" (UID: "9abef49e-37f5-442a-9881-769d93359722"). InnerVolumeSpecName "kube-api-access-sp9sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:39.477206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.477162 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9abef49e-37f5-442a-9881-769d93359722-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:39.477206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.477196 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp9sj\" (UniqueName: \"kubernetes.io/projected/9abef49e-37f5-442a-9881-769d93359722-kube-api-access-sp9sj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:39.477206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:39.477207 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9abef49e-37f5-442a-9881-769d93359722-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:40.115401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:40.115370 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" Nov 02 17:38:40.115583 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:40.115368 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv" event={"ID":"9abef49e-37f5-442a-9881-769d93359722","Type":"ContainerDied","Data":"fac320d032e44f1059236d05749845bc4e2fd4333b5d768197f545547b8e036c"} Nov 02 17:38:40.115583 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:40.115504 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac320d032e44f1059236d05749845bc4e2fd4333b5d768197f545547b8e036c" Nov 02 17:38:46.339335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.339241 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x"] Nov 02 17:38:46.339929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.339909 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9abef49e-37f5-442a-9881-769d93359722" containerName="revoke-keys" Nov 02 17:38:46.340027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.339931 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abef49e-37f5-442a-9881-769d93359722" containerName="revoke-keys" Nov 02 17:38:46.340091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.340064 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abef49e-37f5-442a-9881-769d93359722" containerName="revoke-keys" Nov 02 17:38:46.343102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.343081 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.399623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.399592 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x"] Nov 02 17:38:46.447438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.447403 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.447438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.447439 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8td\" (UniqueName: \"kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.447657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.447542 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:38:46.447657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.447611 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:38:46.447720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.447657 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.549295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.549261 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.549468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.549316 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.549468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.549334 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8td\" (UniqueName: \"kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.550026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.550007 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.551964 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.551945 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.607728 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.607656 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8td\" (UniqueName: \"kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td\") pod \"maas-api-revoke-sub-a9aee483-2cf8x\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:46.655419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:46.655384 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:47.034903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:47.034877 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x"] Nov 02 17:38:47.036237 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:38:47.036209 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c37c61_8bd9_4cbe_906a_bd1df943a1fb.slice/crio-a22027b0c97b391aaa5825fdac68a80d7fe6e38b3f16e9d1c10ec3d0052ff260 WatchSource:0}: Error finding container a22027b0c97b391aaa5825fdac68a80d7fe6e38b3f16e9d1c10ec3d0052ff260: Status 404 returned error can't find the container with id a22027b0c97b391aaa5825fdac68a80d7fe6e38b3f16e9d1c10ec3d0052ff260 Nov 02 17:38:47.155150 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:47.155111 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" event={"ID":"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb","Type":"ContainerStarted","Data":"a22027b0c97b391aaa5825fdac68a80d7fe6e38b3f16e9d1c10ec3d0052ff260"} Nov 02 17:38:48.162774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:48.162652 5647 generic.go:356] "Generic (PLEG): container finished" podID="a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" containerID="c36a0b9988e475997842d3db060c75987bc1699ecaf7de062f10204efe948050" exitCode=0 Nov 02 17:38:48.162774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:48.162727 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" event={"ID":"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb","Type":"ContainerDied","Data":"c36a0b9988e475997842d3db060c75987bc1699ecaf7de062f10204efe948050"} Nov 02 17:38:49.309830 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.309804 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:49.479996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.479896 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca\") pod \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " Nov 02 17:38:49.479996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.479953 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca\") pod \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " Nov 02 17:38:49.480230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.480133 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td\" (UniqueName: \"kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td\") pod \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\" (UID: \"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb\") " Nov 02 17:38:49.480327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.480303 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca" pod "a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" (UID: "a0c37c61-8bd9-4cbe-906a-bd1df943a1fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:38:49.480522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.480500 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:49.482157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.482130 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca" pod "a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" (UID: "a0c37c61-8bd9-4cbe-906a-bd1df943a1fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:38:49.482257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.482219 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td" pod "a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" (UID: "a0c37c61-8bd9-4cbe-906a-bd1df943a1fb"). InnerVolumeSpecName "kube-api-access-2n8td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:38:49.582015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.581983 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n8td\" (UniqueName: \"kubernetes.io/projected/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-kube-api-access-2n8td\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:49.582015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:49.582009 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:38:50.175369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:50.175334 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" Nov 02 17:38:50.175369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:50.175365 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x" event={"ID":"a0c37c61-8bd9-4cbe-906a-bd1df943a1fb","Type":"ContainerDied","Data":"a22027b0c97b391aaa5825fdac68a80d7fe6e38b3f16e9d1c10ec3d0052ff260"} Nov 02 17:38:50.175589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:50.175411 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22027b0c97b391aaa5825fdac68a80d7fe6e38b3f16e9d1c10ec3d0052ff260" Nov 02 17:38:50.604363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:38:50.604327 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:39:05.945908 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:39:05.945867 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:39:08.062220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.062184 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9"] Nov 02 17:39:08.062656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.062629 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" containerName="revoke-keys" Nov 02 17:39:08.062700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.062657 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" containerName="revoke-keys" Nov 02 17:39:08.062759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.062753 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" containerName="revoke-keys" Nov 02 17:39:08.066006 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.065990 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.120612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.120560 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f80493dfdd44\"" Nov 02 17:39:08.120839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.120625 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 17:39:08.171749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.171697 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 17:39:08.171749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.171697 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 17:39:08.171977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.171768 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-cvktf\"" Nov 02 17:39:08.171977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.171769 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 17:39:08.171977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.171787 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2296d33cdbd2\"" Nov 02 17:39:08.173688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.173668 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9"] Nov 02 17:39:08.203412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.203381 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.203607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.203457 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.203701 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.203684 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwc7\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.203779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.203715 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.305000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.304962 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.305201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.305036 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.305201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.305082 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwc7\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.305201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.305110 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.305718 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.305690 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.306016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.305809 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.307692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.307666 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.364095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.364063 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwc7\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7\") pod \"payload-processing-external-model-d98cf8fdf-6r5w9\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.378982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.378953 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:08.760368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:08.760338 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9"] Nov 02 17:39:08.761420 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:08.761380 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596778e9_7a09_4bc8_9ca2_57a731c1e897.slice/crio-2c8da992d98457e633a1cd79ef5f9b2ed5c94b6fbdb50f5167f7cac1192e2462 WatchSource:0}: Error finding container 2c8da992d98457e633a1cd79ef5f9b2ed5c94b6fbdb50f5167f7cac1192e2462: Status 404 returned error can't find the container with id 2c8da992d98457e633a1cd79ef5f9b2ed5c94b6fbdb50f5167f7cac1192e2462 Nov 02 17:39:09.289790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:09.289728 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" event={"ID":"596778e9-7a09-4bc8-9ca2-57a731c1e897","Type":"ContainerStarted","Data":"8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518"} Nov 02 17:39:09.289790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:09.289792 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" event={"ID":"596778e9-7a09-4bc8-9ca2-57a731c1e897","Type":"ContainerStarted","Data":"2c8da992d98457e633a1cd79ef5f9b2ed5c94b6fbdb50f5167f7cac1192e2462"} Nov 02 17:39:09.290215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:09.289822 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:09.408204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:09.408139 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" podStartSLOduration=1.408114456 podStartE2EDuration="1.408114456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:09.406913491 +0000 UTC m=+936.480440698" watchObservedRunningTime="2026-11-02 17:39:09.408114456 +0000 UTC m=+936.481641663" Nov 02 17:39:09.979212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:09.979176 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c"] Nov 02 17:39:09.982967 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:09.982949 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.038130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.038083 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:39:10.038130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.038114 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:39:10.041192 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.041154 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c"] Nov 02 17:39:10.130430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.130392 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.130430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.130432 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mn7\" (UniqueName: \"kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.130667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.130516 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.231598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.231511 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.231598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.231551 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mn7\" (UniqueName: \"kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.231878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.231754 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.232399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.232381 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.234045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.234026 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.289692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.289653 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mn7\" (UniqueName: \"kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7\") pod \"maas-api-revoke-sub-69aaa5bf-hl85c\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.296251 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.296227 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:10.669287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:10.669261 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c"] Nov 02 17:39:10.671058 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:10.671025 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded97dbd4_d222_4d18_8250_c9055487ae40.slice/crio-739f5bdddcae18a39668b9737bafa8634fd663608af5029e3a4a8105d71f99a3 WatchSource:0}: Error finding container 739f5bdddcae18a39668b9737bafa8634fd663608af5029e3a4a8105d71f99a3: Status 404 returned error can't find the container with id 739f5bdddcae18a39668b9737bafa8634fd663608af5029e3a4a8105d71f99a3 Nov 02 17:39:11.304548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:11.304461 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" event={"ID":"ed97dbd4-d222-4d18-8250-c9055487ae40","Type":"ContainerStarted","Data":"836c82859462236f2eef4faeef705e99c317d61374a69780dd5b64d8cd0c59a6"} Nov 02 17:39:11.304548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:11.304536 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" event={"ID":"ed97dbd4-d222-4d18-8250-c9055487ae40","Type":"ContainerStarted","Data":"739f5bdddcae18a39668b9737bafa8634fd663608af5029e3a4a8105d71f99a3"} Nov 02 17:39:11.422599 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:11.422543 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" podStartSLOduration=2.422530795 podStartE2EDuration="2.422530795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:11.420603229 +0000 UTC m=+938.494130432" watchObservedRunningTime="2026-11-02 17:39:11.422530795 +0000 UTC m=+938.496057998" Nov 02 17:39:12.312665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:12.312623 5647 generic.go:356] "Generic (PLEG): container finished" podID="ed97dbd4-d222-4d18-8250-c9055487ae40" containerID="836c82859462236f2eef4faeef705e99c317d61374a69780dd5b64d8cd0c59a6" exitCode=0 Nov 02 17:39:12.313060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:12.312667 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" event={"ID":"ed97dbd4-d222-4d18-8250-c9055487ae40","Type":"ContainerDied","Data":"836c82859462236f2eef4faeef705e99c317d61374a69780dd5b64d8cd0c59a6"} Nov 02 17:39:13.453355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.453328 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:13.568715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.568621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7\" (UniqueName: \"kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7\") pod \"ed97dbd4-d222-4d18-8250-c9055487ae40\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " Nov 02 17:39:13.568907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.568814 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca\") pod \"ed97dbd4-d222-4d18-8250-c9055487ae40\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " Nov 02 17:39:13.568907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.568865 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca\") pod \"ed97dbd4-d222-4d18-8250-c9055487ae40\" (UID: \"ed97dbd4-d222-4d18-8250-c9055487ae40\") " Nov 02 17:39:13.569539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.569512 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca" pod "ed97dbd4-d222-4d18-8250-c9055487ae40" (UID: "ed97dbd4-d222-4d18-8250-c9055487ae40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:39:13.571390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.571363 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7" pod "ed97dbd4-d222-4d18-8250-c9055487ae40" (UID: "ed97dbd4-d222-4d18-8250-c9055487ae40"). InnerVolumeSpecName "kube-api-access-64mn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:39:13.571508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.571459 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca" pod "ed97dbd4-d222-4d18-8250-c9055487ae40" (UID: "ed97dbd4-d222-4d18-8250-c9055487ae40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:39:13.670272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.670229 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:39:13.670272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.670263 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64mn7\" (UniqueName: \"kubernetes.io/projected/ed97dbd4-d222-4d18-8250-c9055487ae40-kube-api-access-64mn7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:39:13.670272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:13.670275 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed97dbd4-d222-4d18-8250-c9055487ae40-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:39:14.325378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:14.325351 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" Nov 02 17:39:14.325378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:14.325371 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c" event={"ID":"ed97dbd4-d222-4d18-8250-c9055487ae40","Type":"ContainerDied","Data":"739f5bdddcae18a39668b9737bafa8634fd663608af5029e3a4a8105d71f99a3"} Nov 02 17:39:14.325596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:14.325420 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="739f5bdddcae18a39668b9737bafa8634fd663608af5029e3a4a8105d71f99a3" Nov 02 17:39:16.411466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:16.411422 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:39:17.483833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.483798 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6"] Nov 02 17:39:17.484260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.484245 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ed97dbd4-d222-4d18-8250-c9055487ae40" containerName="revoke-keys" Nov 02 17:39:17.484305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.484260 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed97dbd4-d222-4d18-8250-c9055487ae40" containerName="revoke-keys" Nov 02 17:39:17.484348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.484339 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed97dbd4-d222-4d18-8250-c9055487ae40" containerName="revoke-keys" Nov 02 17:39:17.487402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.487386 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.563063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.563027 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6"] Nov 02 17:39:17.593282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.593246 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"openshift-service-ca.crt\"" Nov 02 17:39:17.593469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.593389 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"e2e-conflict-a-562d66ee-kserve-self-signed-certs\"" Nov 02 17:39:17.593469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.593428 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"kube-root-ca.crt\"" Nov 02 17:39:17.593608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.593474 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"default-dockercfg-2jpz8\"" Nov 02 17:39:17.621836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.621640 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42nhn\" (UniqueName: \"kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.621836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.621793 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.621836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.621829 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.622141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.621929 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.622141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.621967 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.622141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.621988 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.723428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.723391 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42nhn\" (UniqueName: \"kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.723606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.723481 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.723606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.723509 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.723606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.723584 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.723846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.723609 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.723846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.723632 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.724118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.724088 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.724254 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.724148 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.724254 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.724216 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.726333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.726297 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.726443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.726341 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.782166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.782076 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42nhn\" (UniqueName: \"kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn\") pod \"e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:17.800528 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:17.800500 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:18.215223 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:18.215186 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6"] Nov 02 17:39:18.217229 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:18.217185 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd41c0d6b_7bdf_48ea_bf44_860854190ae3.slice/crio-d424cf93fff316f5088f12086c344a497897ecc461aaa4d97d93bc006998d88f WatchSource:0}: Error finding container d424cf93fff316f5088f12086c344a497897ecc461aaa4d97d93bc006998d88f: Status 404 returned error can't find the container with id d424cf93fff316f5088f12086c344a497897ecc461aaa4d97d93bc006998d88f Nov 02 17:39:18.354620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:18.354568 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" event={"ID":"d41c0d6b-7bdf-48ea-bf44-860854190ae3","Type":"ContainerStarted","Data":"920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb"} Nov 02 17:39:18.354620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:18.354622 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" event={"ID":"d41c0d6b-7bdf-48ea-bf44-860854190ae3","Type":"ContainerStarted","Data":"d424cf93fff316f5088f12086c344a497897ecc461aaa4d97d93bc006998d88f"} Nov 02 17:39:18.474275 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:18.474153 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" podStartSLOduration=1.474133426 podStartE2EDuration="1.474133426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:18.471943345 +0000 UTC m=+945.545470553" watchObservedRunningTime="2026-11-02 17:39:18.474133426 +0000 UTC m=+945.547660632" Nov 02 17:39:18.801610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:18.801529 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:20.299492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:20.299464 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:39:21.976863 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:21.976828 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw"] Nov 02 17:39:21.980628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:21.980606 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080336 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080303 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080642 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080717 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080801 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080854 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbgw\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080969 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080929 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.080969 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080957 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.081088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.080983 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.081088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.081010 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.081913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.081889 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw"] Nov 02 17:39:22.182238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182203 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182269 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182345 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182389 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbgw\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182454 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182478 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182505 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.182680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.182534 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.183050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.183020 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.183050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.183046 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.183249 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.183194 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.183867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.183829 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.184061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.184041 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.185281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.185254 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.185496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.185475 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.240631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.240548 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.240805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.240631 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbgw\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqqgw\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.294471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.294435 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:22.723337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.723313 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw"] Nov 02 17:39:22.724848 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:22.724820 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9de4197_e8e0_4abd_a991_d3d9e89e4b45.slice/crio-18b7189ecd19c360981c0f8e001f27f18603d9510369df39132768c232c3da98 WatchSource:0}: Error finding container 18b7189ecd19c360981c0f8e001f27f18603d9510369df39132768c232c3da98: Status 404 returned error can't find the container with id 18b7189ecd19c360981c0f8e001f27f18603d9510369df39132768c232c3da98 Nov 02 17:39:22.727345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.727310 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:22.727432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.727399 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:22.727432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:22.727429 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:23.387915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:23.387877 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" event={"ID":"c9de4197-e8e0-4abd-a991-d3d9e89e4b45","Type":"ContainerStarted","Data":"7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3"} Nov 02 17:39:23.388384 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:23.387935 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" event={"ID":"c9de4197-e8e0-4abd-a991-d3d9e89e4b45","Type":"ContainerStarted","Data":"18b7189ecd19c360981c0f8e001f27f18603d9510369df39132768c232c3da98"} Nov 02 17:39:23.587035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:23.586984 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podStartSLOduration=2.586968274 podStartE2EDuration="2.586968274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:23.559066026 +0000 UTC m=+950.632593255" watchObservedRunningTime="2026-11-02 17:39:23.586968274 +0000 UTC m=+950.660495476" Nov 02 17:39:24.295573 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:24.295529 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:24.297247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:24.297221 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:24.297363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:24.297270 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:25.294982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:25.294950 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:25.295379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:25.295001 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:26.295185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:26.295148 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:26.295576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:26.295214 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:27.294908 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:27.294874 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:27.295081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:27.294940 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:27.802780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:27.800759 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:27.859224 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:27.859193 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:28.295610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:28.295571 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:28.295835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:28.295646 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:28.448474 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:28.448446 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:39:29.295494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:29.295452 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:29.296101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:29.295509 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:30.295570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:30.295524 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Nov 02 17:39:30.296153 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:30.295582 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Nov 02 17:39:31.252567 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.252533 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx"] Nov 02 17:39:31.260777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.260712 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.308382 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.308341 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mgk\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.308382 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.308388 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.308976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.308411 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.308976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.308558 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.312594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.312569 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx"] Nov 02 17:39:31.410078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.410038 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.410078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.410079 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.410337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.410204 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.410337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.410319 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mgk\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.410923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.410904 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.411122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.411104 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.413449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.413361 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.413601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.413550 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:31.414952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.414217 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:31.415090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.415050 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:39:31.469139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.469108 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mgk\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk\") pod \"payload-processing-external-model-6c67f497c-fwpxx\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.579327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.579239 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:31.956046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:31.956017 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx"] Nov 02 17:39:31.957808 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:31.957782 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7151da_b961_4d5d_a982_1054a13e49bd.slice/crio-b710bb50456989f5c6963e7eaf3678ad2318eb7018277bf444846da99676fd7a WatchSource:0}: Error finding container b710bb50456989f5c6963e7eaf3678ad2318eb7018277bf444846da99676fd7a: Status 404 returned error can't find the container with id b710bb50456989f5c6963e7eaf3678ad2318eb7018277bf444846da99676fd7a Nov 02 17:39:32.462694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:32.462661 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" event={"ID":"8d7151da-b961-4d5d-a982-1054a13e49bd","Type":"ContainerStarted","Data":"d32b304e1776d9897b5bb54ccee1979b45976a8a30fcc4b7832b532e0fcdf923"} Nov 02 17:39:32.463220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:32.462710 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" event={"ID":"8d7151da-b961-4d5d-a982-1054a13e49bd","Type":"ContainerStarted","Data":"b710bb50456989f5c6963e7eaf3678ad2318eb7018277bf444846da99676fd7a"} Nov 02 17:39:32.463220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:32.462984 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:39:32.580413 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:32.580359 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podStartSLOduration=1.580343933 podStartE2EDuration="1.580343933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:32.57922477 +0000 UTC m=+959.652751972" watchObservedRunningTime="2026-11-02 17:39:32.580343933 +0000 UTC m=+959.653871183" Nov 02 17:39:35.947026 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:39:35.946983 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:39:37.096634 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.096605 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs"] Nov 02 17:39:37.100603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.100585 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.147468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.147433 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk"] Nov 02 17:39:37.151308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.151291 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.175200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175134 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175183 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175290 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175319 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fp6p\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175350 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175453 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175547 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175664 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.175830 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.175699 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.202321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.202289 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs"] Nov 02 17:39:37.260275 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.260246 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk"] Nov 02 17:39:37.276686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.276653 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.276877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.276752 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.276877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.276819 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.276877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.276863 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.276882 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.276972 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277005 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277041 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277071 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277204 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277232 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277245 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277261 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277287 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277311 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp6p\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277360 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277385 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277429 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277477 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv598\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277507 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277521 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277612 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.277987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.277965 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.279622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.279598 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.279851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.279834 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.339920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.339878 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.340106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.340066 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp6p\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-btngs\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.379010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.378927 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.378981 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379024 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv598\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379107 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379129 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379202 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379237 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379273 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379302 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379442 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379759 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379720 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.379880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379833 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.380004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.379977 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.381971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.381946 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.382149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.382126 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.413055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.413026 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:37.438474 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.438448 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.438600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.438581 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv598\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-6t4wk\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.465142 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.464269 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:37.839807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.839778 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs"] Nov 02 17:39:37.841002 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:37.840970 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9aa48f_d3bf_434a_aca7_e1f52c0b68e9.slice/crio-f5ce669512ce7bcbad7f64fb65dbda72034c0daf485d61552d81e10db280b132 WatchSource:0}: Error finding container f5ce669512ce7bcbad7f64fb65dbda72034c0daf485d61552d81e10db280b132: Status 404 returned error can't find the container with id f5ce669512ce7bcbad7f64fb65dbda72034c0daf485d61552d81e10db280b132 Nov 02 17:39:37.843537 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.843504 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:37.843635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.843570 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:37.843635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.843601 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:37.892552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.892529 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk"] Nov 02 17:39:37.893524 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:37.893500 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f266285_791c_467e_aba7_801f8203f39a.slice/crio-f055eefbb43d7b99573eab8e23bfc9d5c7e787250d634cf1073650c96407c96c WatchSource:0}: Error finding container f055eefbb43d7b99573eab8e23bfc9d5c7e787250d634cf1073650c96407c96c: Status 404 returned error can't find the container with id f055eefbb43d7b99573eab8e23bfc9d5c7e787250d634cf1073650c96407c96c Nov 02 17:39:37.896141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.896105 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:37.896220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.896175 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:37.896220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:37.896205 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:38.499516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:38.499483 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" event={"ID":"8f266285-791c-467e-aba7-801f8203f39a","Type":"ContainerStarted","Data":"f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3"} Nov 02 17:39:38.500113 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:38.499539 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" event={"ID":"8f266285-791c-467e-aba7-801f8203f39a","Type":"ContainerStarted","Data":"f055eefbb43d7b99573eab8e23bfc9d5c7e787250d634cf1073650c96407c96c"} Nov 02 17:39:38.501276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:38.501252 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" event={"ID":"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9","Type":"ContainerStarted","Data":"934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300"} Nov 02 17:39:38.501442 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:38.501304 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" event={"ID":"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9","Type":"ContainerStarted","Data":"f5ce669512ce7bcbad7f64fb65dbda72034c0daf485d61552d81e10db280b132"} Nov 02 17:39:38.645761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:38.643264 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podStartSLOduration=1.6432439269999999 podStartE2EDuration="1.643243927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:38.641367749 +0000 UTC m=+965.714894952" watchObservedRunningTime="2026-11-02 17:39:38.643243927 +0000 UTC m=+965.716771132" Nov 02 17:39:38.763837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:38.763713 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podStartSLOduration=1.763696936 podStartE2EDuration="1.763696936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:38.761071922 +0000 UTC m=+965.834599126" watchObservedRunningTime="2026-11-02 17:39:38.763696936 +0000 UTC m=+965.837224142" Nov 02 17:39:39.422759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:39.422127 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:39.432770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:39.431871 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:39.432770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:39.431938 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:39.465321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:39.465287 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:39.466704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:39.466679 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:39.466814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:39.466744 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:40.421759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:40.421142 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:40.421759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:40.421202 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:40.465255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:40.465221 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:40.465436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:40.465271 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:41.416212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:41.416169 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:41.416534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:41.416501 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:41.465477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:41.465445 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:41.465924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:41.465900 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:42.199541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:42.199508 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:39:42.414229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:42.414190 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:42.414522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:42.414491 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:42.465774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:42.465582 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:42.465774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:42.465642 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:43.417200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:43.417159 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:43.417993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:43.417519 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:43.465592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:43.465554 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:43.465914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:43.465885 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:43.474757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:43.474606 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:39:44.414179 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:44.414136 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:44.414459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:44.414429 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:44.465125 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:44.465088 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:44.465748 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:44.465695 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:45.415203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:45.415158 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Nov 02 17:39:45.415871 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:45.415836 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Nov 02 17:39:45.466156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:45.466117 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Nov 02 17:39:45.466465 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:45.466432 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Nov 02 17:39:47.413769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:47.413711 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:39:47.414341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:47.413801 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:39:47.466075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:47.466037 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:39:47.466364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:47.466328 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:39:48.454907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:48.454869 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:39:48.455556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:48.455524 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:39:48.507915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:48.507877 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:39:48.508212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:48.508182 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:39:49.497916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.497875 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:39:49.498435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.497934 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:39:49.550957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.550913 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:39:49.551257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.551228 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:39:49.748224 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.748128 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:49.748477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.748453 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:49.749246 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.749226 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:39:49.894124 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.894092 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:49.894439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.894408 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:49.895111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:49.895092 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:39:50.085950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.085901 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf"] Nov 02 17:39:50.089884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.089860 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.148332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.148304 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a099c3-openshift-default-dockercfg-68w8k\"" Nov 02 17:39:50.149997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.149963 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shcr\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150188 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150258 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150416 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150443 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150474 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150521 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150592 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.150639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.150625 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.165094 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.165067 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf"] Nov 02 17:39:50.252004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.251959 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252158 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252025 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252158 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252135 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shcr\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252168 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252220 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252321 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252349 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252387 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252426 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252573 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252772 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252805 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252844 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.252895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.252869 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.254687 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.254663 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.255008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.254983 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.310668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.310635 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.310888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.310870 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shcr\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.405935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.405827 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:50.836262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.836233 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf"] Nov 02 17:39:50.837773 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:39:50.837727 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b906cd3_3ec1_4228_97f6_5de992a1aa81.slice/crio-c079868c2eaf78599472d8a1bb901bda194e3092f0a8421654b23d8aab4209c5 WatchSource:0}: Error finding container c079868c2eaf78599472d8a1bb901bda194e3092f0a8421654b23d8aab4209c5: Status 404 returned error can't find the container with id c079868c2eaf78599472d8a1bb901bda194e3092f0a8421654b23d8aab4209c5 Nov 02 17:39:50.840238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.840206 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:50.840310 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.840286 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:50.840342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:50.840317 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:39:51.653262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:51.653229 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" event={"ID":"9b906cd3-3ec1-4228-97f6-5de992a1aa81","Type":"ContainerStarted","Data":"8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9"} Nov 02 17:39:51.653493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:51.653281 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" event={"ID":"9b906cd3-3ec1-4228-97f6-5de992a1aa81","Type":"ContainerStarted","Data":"c079868c2eaf78599472d8a1bb901bda194e3092f0a8421654b23d8aab4209c5"} Nov 02 17:39:51.830324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:51.830253 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podStartSLOduration=1.83023334 podStartE2EDuration="1.83023334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:51.827954388 +0000 UTC m=+978.901481584" watchObservedRunningTime="2026-11-02 17:39:51.83023334 +0000 UTC m=+978.903760545" Nov 02 17:39:52.410768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:52.410543 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:52.410768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:52.410693 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:39:52.411706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:52.411387 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:39:53.408770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:53.408107 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:39:53.408770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:53.408171 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:39:53.470236 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:53.470200 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:39:54.411762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:54.409034 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:39:54.411762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:54.409107 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:39:55.426887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:55.426849 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:39:55.427332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:55.426908 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:39:56.426912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:56.426856 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:39:56.427457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:56.426938 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:39:57.422221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:57.416163 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Nov 02 17:39:57.422221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:57.416224 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Nov 02 17:39:58.411083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:58.411051 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:58.411548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:58.411341 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:39:58.412061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:39:58.412037 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:40:03.469788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:03.469723 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:40:05.947068 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:05.947029 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:40:08.792578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:08.792536 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:40:13.469540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:13.469447 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:40:23.469835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:23.469788 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:40:30.572906 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.572863 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f"] Nov 02 17:40:30.576801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.576783 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.632640 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.632600 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f"] Nov 02 17:40:30.662084 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.662042 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.662084 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.662088 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.662309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.662108 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th8v2\" (UniqueName: \"kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.662309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.662255 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.662378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.662322 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.662412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.662377 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.681111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.681083 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"e2e-conflict-b-562d66ee-kserve-self-signed-certs\"" Nov 02 17:40:30.764439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.764391 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.767959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.767925 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.768238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.768215 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.769541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.768486 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.769757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.768651 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.769757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.769620 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.769757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.769667 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.769757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.769701 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th8v2\" (UniqueName: \"kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.770085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.770053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.771443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.771423 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.772498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.772449 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.827577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.827492 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th8v2\" (UniqueName: \"kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2\") pod \"e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:30.889543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:30.889501 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:31.319686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:31.319646 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f"] Nov 02 17:40:31.320852 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:40:31.320825 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod538b0f02_cb5e_43b0_bad3_380811961755.slice/crio-0b6b0cfceebc54cc1a06952eb1bdc29511521d53870a744a7bd8999ab6e3e1b9 WatchSource:0}: Error finding container 0b6b0cfceebc54cc1a06952eb1bdc29511521d53870a744a7bd8999ab6e3e1b9: Status 404 returned error can't find the container with id 0b6b0cfceebc54cc1a06952eb1bdc29511521d53870a744a7bd8999ab6e3e1b9 Nov 02 17:40:31.921094 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:31.921054 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" event={"ID":"538b0f02-cb5e-43b0-bad3-380811961755","Type":"ContainerStarted","Data":"fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d"} Nov 02 17:40:31.921512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:31.921104 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" event={"ID":"538b0f02-cb5e-43b0-bad3-380811961755","Type":"ContainerStarted","Data":"0b6b0cfceebc54cc1a06952eb1bdc29511521d53870a744a7bd8999ab6e3e1b9"} Nov 02 17:40:32.039239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:32.039188 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" podStartSLOduration=2.039173042 podStartE2EDuration="2.039173042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:32.037068239 +0000 UTC m=+1019.110595443" watchObservedRunningTime="2026-11-02 17:40:32.039173042 +0000 UTC m=+1019.112700245" Nov 02 17:40:32.197859 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:32.197291 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f80493dfdd44: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:32.197859 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:32.197320 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:32.197859 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:32.197387 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials podName:596778e9-7a09-4bc8-9ca2-57a731c1e897 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:32.69736412 +0000 UTC m=+1019.770891324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials") pod "payload-processing-external-model-d98cf8fdf-6r5w9" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897") : secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:32.702817 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:32.702781 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f80493dfdd44: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:32.702817 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:32.702810 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:32.703033 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:32.702874 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials podName:596778e9-7a09-4bc8-9ca2-57a731c1e897 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:33.702854908 +0000 UTC m=+1020.776382105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials") pod "payload-processing-external-model-d98cf8fdf-6r5w9" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897") : secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:32.709010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:32.708978 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx"] Nov 02 17:40:32.710412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:32.710382 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 17:40:32.889885 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:32.889838 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:33.239641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:33.239612 5647 scope.go:122] "RemoveContainer" containerID="39936686ab79cb7760c6634f4eaafabb9f895177cd55b32ac9ace7bd55647455" Nov 02 17:40:33.251307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:33.251281 5647 scope.go:122] "RemoveContainer" containerID="260545a83f15b3e6806ab9b26f1d6dc3c90c8b6df2d5aae35c9cb2f85a879354" Nov 02 17:40:33.261383 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:33.261361 5647 scope.go:122] "RemoveContainer" containerID="ae92423a132b4598501bd509f1253d611ddf1b020e6d50fcc134e756a40bb109" Nov 02 17:40:33.714064 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:33.714026 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f80493dfdd44: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:33.714064 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:33.714062 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:33.714314 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:33.714136 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials podName:596778e9-7a09-4bc8-9ca2-57a731c1e897 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:35.714113505 +0000 UTC m=+1022.787640687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials") pod "payload-processing-external-model-d98cf8fdf-6r5w9" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897") : secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:35.738541 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:35.738506 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f80493dfdd44: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:35.738541 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:35.738534 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:35.739069 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:35.738602 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials podName:596778e9-7a09-4bc8-9ca2-57a731c1e897 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:39.73858701 +0000 UTC m=+1026.812114205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials") pod "payload-processing-external-model-d98cf8fdf-6r5w9" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897") : secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:35.948680 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:35.948641 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:40:36.940256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:36.940220 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:40:37.736943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:37.730812 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" containerID="cri-o://d32b304e1776d9897b5bb54ccee1979b45976a8a30fcc4b7832b532e0fcdf923" gracePeriod=25 Nov 02 17:40:37.984478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:37.984151 5647 generic.go:356] "Generic (PLEG): container finished" podID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerID="d32b304e1776d9897b5bb54ccee1979b45976a8a30fcc4b7832b532e0fcdf923" exitCode=0 Nov 02 17:40:37.984478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:37.984361 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" event={"ID":"8d7151da-b961-4d5d-a982-1054a13e49bd","Type":"ContainerDied","Data":"d32b304e1776d9897b5bb54ccee1979b45976a8a30fcc4b7832b532e0fcdf923"} Nov 02 17:40:38.117800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.116701 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:40:38.196888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.196800 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials\") pod \"8d7151da-b961-4d5d-a982-1054a13e49bd\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " Nov 02 17:40:38.196888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.196862 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume\") pod \"8d7151da-b961-4d5d-a982-1054a13e49bd\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " Nov 02 17:40:38.197162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.196913 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume\") pod \"8d7151da-b961-4d5d-a982-1054a13e49bd\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " Nov 02 17:40:38.197162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.196957 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk\") pod \"8d7151da-b961-4d5d-a982-1054a13e49bd\" (UID: \"8d7151da-b961-4d5d-a982-1054a13e49bd\") " Nov 02 17:40:38.198782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.197690 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume" pod "8d7151da-b961-4d5d-a982-1054a13e49bd" (UID: "8d7151da-b961-4d5d-a982-1054a13e49bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:38.198782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.197698 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume" pod "8d7151da-b961-4d5d-a982-1054a13e49bd" (UID: "8d7151da-b961-4d5d-a982-1054a13e49bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:38.198782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.198260 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:38.198782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.198283 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8d7151da-b961-4d5d-a982-1054a13e49bd-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:38.199950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.199921 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials" pod "8d7151da-b961-4d5d-a982-1054a13e49bd" (UID: "8d7151da-b961-4d5d-a982-1054a13e49bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:38.201079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.201050 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk" pod "8d7151da-b961-4d5d-a982-1054a13e49bd" (UID: "8d7151da-b961-4d5d-a982-1054a13e49bd"). InnerVolumeSpecName "kube-api-access-h4mgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:38.299399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.299359 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mgk\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-kube-api-access-h4mgk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:38.299399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.299400 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8d7151da-b961-4d5d-a982-1054a13e49bd-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:38.998080 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.998047 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" Nov 02 17:40:38.999263 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.998080 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx" event={"ID":"8d7151da-b961-4d5d-a982-1054a13e49bd","Type":"ContainerDied","Data":"b710bb50456989f5c6963e7eaf3678ad2318eb7018277bf444846da99676fd7a"} Nov 02 17:40:38.999395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:38.999305 5647 scope.go:122] "RemoveContainer" containerID="d32b304e1776d9897b5bb54ccee1979b45976a8a30fcc4b7832b532e0fcdf923" Nov 02 17:40:39.180520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:39.180478 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx"] Nov 02 17:40:39.241759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:39.241565 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6c67f497c-fwpxx"] Nov 02 17:40:39.765156 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:39.765123 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f80493dfdd44: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:39.784784 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:39.776840 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:39.784784 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:39.776994 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials podName:596778e9-7a09-4bc8-9ca2-57a731c1e897 nodeName:}" failed. No retries permitted until 2026-11-02 17:40:47.77696907 +0000 UTC m=+1034.850496266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials") pod "payload-processing-external-model-d98cf8fdf-6r5w9" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897") : secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:40.890405 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:40.890369 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:40.945759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:40.945099 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:41.101344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.101307 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" path="/var/lib/kubelet/pods/8d7151da-b961-4d5d-a982-1054a13e49bd/volumes" Nov 02 17:40:41.101697 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.101678 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:40:41.113771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.112043 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq"] Nov 02 17:40:41.113771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.112559 5647 cpu_manager.go:414] "containerMap: removing container" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" Nov 02 17:40:41.113771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.112575 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" Nov 02 17:40:41.113771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.112757 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7151da-b961-4d5d-a982-1054a13e49bd" containerName="payload-processing" Nov 02 17:40:41.120873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.118514 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.172139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.172049 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d4530acdcb45\"" Nov 02 17:40:41.172139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.172057 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7981791ff29e\"" Nov 02 17:40:41.183507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.183473 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq"] Nov 02 17:40:41.226452 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.226417 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jt2\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.226660 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.226467 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.226727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.226650 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.226819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.226796 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.328518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.328486 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.328769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.328563 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.328769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.328677 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jt2\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.328769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.328713 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.329590 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.329416 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.329590 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.329504 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.331810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.331785 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.386861 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.386827 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jt2\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2\") pod \"payload-processing-external-model-7599585dbd-lsvnq\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.438310 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.438217 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:41.814039 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:41.814008 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq"] Nov 02 17:40:41.815611 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:40:41.815586 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc162bf2_1da3_4a5e_880b_3f9230e5e2ec.slice/crio-1afa25d50f881b9fd366264e7cbb52cce1a6135a8139e1cff012f78eeb37048d WatchSource:0}: Error finding container 1afa25d50f881b9fd366264e7cbb52cce1a6135a8139e1cff012f78eeb37048d: Status 404 returned error can't find the container with id 1afa25d50f881b9fd366264e7cbb52cce1a6135a8139e1cff012f78eeb37048d Nov 02 17:40:42.057137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:42.057091 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" event={"ID":"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec","Type":"ContainerStarted","Data":"88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9"} Nov 02 17:40:42.057582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:42.057162 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" event={"ID":"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec","Type":"ContainerStarted","Data":"1afa25d50f881b9fd366264e7cbb52cce1a6135a8139e1cff012f78eeb37048d"} Nov 02 17:40:42.057582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:42.057372 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:42.174307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:42.174253 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" podStartSLOduration=1.174238971 podStartE2EDuration="1.174238971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:42.172953576 +0000 UTC m=+1029.246480783" watchObservedRunningTime="2026-11-02 17:40:42.174238971 +0000 UTC m=+1029.247766191" Nov 02 17:40:47.802543 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:47.802506 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f80493dfdd44: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:47.802543 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:47.802536 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9: secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:47.803010 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:47.802596 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials podName:596778e9-7a09-4bc8-9ca2-57a731c1e897 nodeName:}" failed. No retries permitted until 2026-11-02 17:41:03.8025779 +0000 UTC m=+1050.876105097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials") pod "payload-processing-external-model-d98cf8fdf-6r5w9" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897") : secret "agc-cred-f80493dfdd44" not found Nov 02 17:40:53.065901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:53.065867 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:40:53.234794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:53.234761 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9"] Nov 02 17:40:58.235690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.235626 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" podUID="596778e9-7a09-4bc8-9ca2-57a731c1e897" containerName="payload-processing" containerID="cri-o://8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518" gracePeriod=25 Nov 02 17:40:58.502230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.502164 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:40:58.534090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.534057 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7\") pod \"596778e9-7a09-4bc8-9ca2-57a731c1e897\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " Nov 02 17:40:58.534264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.534130 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials\") pod \"596778e9-7a09-4bc8-9ca2-57a731c1e897\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " Nov 02 17:40:58.534264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.534176 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume\") pod \"596778e9-7a09-4bc8-9ca2-57a731c1e897\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " Nov 02 17:40:58.534407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.534302 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume\") pod \"596778e9-7a09-4bc8-9ca2-57a731c1e897\" (UID: \"596778e9-7a09-4bc8-9ca2-57a731c1e897\") " Nov 02 17:40:58.534661 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.534630 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume" pod "596778e9-7a09-4bc8-9ca2-57a731c1e897" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:58.534781 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.534721 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume" pod "596778e9-7a09-4bc8-9ca2-57a731c1e897" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:40:58.536300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.536276 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials" pod "596778e9-7a09-4bc8-9ca2-57a731c1e897" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:58.536401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.536370 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7" pod "596778e9-7a09-4bc8-9ca2-57a731c1e897" (UID: "596778e9-7a09-4bc8-9ca2-57a731c1e897"). InnerVolumeSpecName "kube-api-access-txwc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:40:58.650757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.642418 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwc7\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-kube-api-access-txwc7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:58.650757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.642458 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/596778e9-7a09-4bc8-9ca2-57a731c1e897-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:58.650757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.642472 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:58.650757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:58.642486 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/596778e9-7a09-4bc8-9ca2-57a731c1e897-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:40:59.171559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.171518 5647 generic.go:356] "Generic (PLEG): container finished" podID="596778e9-7a09-4bc8-9ca2-57a731c1e897" containerID="8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518" exitCode=0 Nov 02 17:40:59.171813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.171703 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" event={"ID":"596778e9-7a09-4bc8-9ca2-57a731c1e897","Type":"ContainerDied","Data":"8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518"} Nov 02 17:40:59.171813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.171803 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" event={"ID":"596778e9-7a09-4bc8-9ca2-57a731c1e897","Type":"ContainerDied","Data":"2c8da992d98457e633a1cd79ef5f9b2ed5c94b6fbdb50f5167f7cac1192e2462"} Nov 02 17:40:59.171972 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.171849 5647 scope.go:122] "RemoveContainer" containerID="8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518" Nov 02 17:40:59.172058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.172040 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9" Nov 02 17:40:59.203817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.203784 5647 scope.go:122] "RemoveContainer" containerID="8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518" Nov 02 17:40:59.204324 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:40:59.204286 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518\": container with ID starting with 8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518 not found: ID does not exist" containerID="8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518" Nov 02 17:40:59.204462 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.204339 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518"} err="failed to get container status \"8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518\": rpc error: code = NotFound desc = could not find container \"8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518\": container with ID starting with 8a65b644e870ff92672e38ff8263b881b15069016dc3ebd70726d97d2f092518 not found: ID does not exist" Nov 02 17:40:59.341524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.341486 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9"] Nov 02 17:40:59.392169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:40:59.392072 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d98cf8fdf-6r5w9"] Nov 02 17:41:01.095719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:01.095658 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596778e9-7a09-4bc8-9ca2-57a731c1e897" path="/var/lib/kubelet/pods/596778e9-7a09-4bc8-9ca2-57a731c1e897/volumes" Nov 02 17:41:05.946285 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:41:05.946242 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:41:07.021194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.021155 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg"] Nov 02 17:41:07.021692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.021678 5647 cpu_manager.go:414] "containerMap: removing container" podUID="596778e9-7a09-4bc8-9ca2-57a731c1e897" containerName="payload-processing" Nov 02 17:41:07.021777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.021693 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596778e9-7a09-4bc8-9ca2-57a731c1e897" containerName="payload-processing" Nov 02 17:41:07.021819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.021797 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="596778e9-7a09-4bc8-9ca2-57a731c1e897" containerName="payload-processing" Nov 02 17:41:07.026242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.026223 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.082177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.082144 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg"] Nov 02 17:41:07.102648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.102610 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.102855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.102652 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7mlh\" (UniqueName: \"kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.102855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.102806 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.102949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.102918 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.102982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.102953 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.102982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.102972 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.129387 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.129328 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"openshift-service-ca.crt\"" Nov 02 17:41:07.129387 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.129371 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"kube-root-ca.crt\"" Nov 02 17:41:07.130260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.130235 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"default-dockercfg-gsl42\"" Nov 02 17:41:07.130383 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.130262 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dc5689\"/\"distinct-2-w1-dc5689-kserve-self-signed-certs\"" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.207878 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208023 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208072 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208096 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208194 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208228 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7mlh\" (UniqueName: \"kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.208793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208585 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.209305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.208935 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.209305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.209016 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.229333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.229297 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.231289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.231263 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.266945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.266914 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7mlh\" (UniqueName: \"kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh\") pod \"distinct-2-w1-dc5689-kserve-bf79c9554-v88zg\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.338778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.338717 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:07.768681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:07.768656 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg"] Nov 02 17:41:07.769868 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:41:07.769844 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b4c3c62_35ca_4a21_88fe_b850c7f63f6d.slice/crio-aeaa4b91405a258dd198b1406a9d7b562d8b057e9523d40dbfa2bde7397bbcfc WatchSource:0}: Error finding container aeaa4b91405a258dd198b1406a9d7b562d8b057e9523d40dbfa2bde7397bbcfc: Status 404 returned error can't find the container with id aeaa4b91405a258dd198b1406a9d7b562d8b057e9523d40dbfa2bde7397bbcfc Nov 02 17:41:08.415783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:08.415709 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" event={"ID":"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d","Type":"ContainerStarted","Data":"c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4"} Nov 02 17:41:08.415783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:08.415787 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" event={"ID":"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d","Type":"ContainerStarted","Data":"aeaa4b91405a258dd198b1406a9d7b562d8b057e9523d40dbfa2bde7397bbcfc"} Nov 02 17:41:08.532789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:08.532715 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podStartSLOduration=2.532700987 podStartE2EDuration="2.532700987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:08.531487713 +0000 UTC m=+1055.605014929" watchObservedRunningTime="2026-11-02 17:41:08.532700987 +0000 UTC m=+1055.606228189" Nov 02 17:41:09.339446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:09.339408 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:15.045925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:15.045800 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:41:17.339415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:17.339378 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:17.352277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:17.352240 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:17.485367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:17.485334 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:41:33.291808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:33.291772 5647 scope.go:122] "RemoveContainer" containerID="52c26eb5283de798db901194f463cfcab4f86c78c559f955b6911de695196666" Nov 02 17:41:33.301144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:33.301119 5647 scope.go:122] "RemoveContainer" containerID="cafab92e2ef7a3a896f15a54c295682ece5398a167d538ea01a45c7880fcf17c" Nov 02 17:41:33.309727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:33.309706 5647 scope.go:122] "RemoveContainer" containerID="42af5ebe4b96c28ecc6548a0e398ee53c8b32cdf0dbe69b24a07ead6b2f624e9" Nov 02 17:41:35.947608 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:41:35.947557 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:41:50.926355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:50.926276 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:41:53.914187 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:41:53.914152 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f"] Nov 02 17:42:05.604708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:05.604677 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6"] Nov 02 17:42:05.947479 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:42:05.947378 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:42:08.934985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:08.934942 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" podUID="538b0f02-cb5e-43b0-bad3-380811961755" containerName="main" containerID="cri-o://fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d" gracePeriod=45 Nov 02 17:42:09.049857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.049807 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" podUID="538b0f02-cb5e-43b0-bad3-380811961755" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused" Nov 02 17:42:09.204887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.204862 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:42:09.286170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286135 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs\" (UniqueName: \"kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs\") pod \"538b0f02-cb5e-43b0-bad3-380811961755\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " Nov 02 17:42:09.286334 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286190 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2\" (UniqueName: \"kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2\") pod \"538b0f02-cb5e-43b0-bad3-380811961755\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " Nov 02 17:42:09.286334 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286244 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache\") pod \"538b0f02-cb5e-43b0-bad3-380811961755\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " Nov 02 17:42:09.286450 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286359 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir\") pod \"538b0f02-cb5e-43b0-bad3-380811961755\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " Nov 02 17:42:09.286508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286477 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm\") pod \"538b0f02-cb5e-43b0-bad3-380811961755\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " Nov 02 17:42:09.286563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286527 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home\") pod \"538b0f02-cb5e-43b0-bad3-380811961755\" (UID: \"538b0f02-cb5e-43b0-bad3-380811961755\") " Nov 02 17:42:09.286618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286570 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache" pod "538b0f02-cb5e-43b0-bad3-380811961755" (UID: "538b0f02-cb5e-43b0-bad3-380811961755"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:09.286618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286585 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir" pod "538b0f02-cb5e-43b0-bad3-380811961755" (UID: "538b0f02-cb5e-43b0-bad3-380811961755"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:09.286860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286832 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home" pod "538b0f02-cb5e-43b0-bad3-380811961755" (UID: "538b0f02-cb5e-43b0-bad3-380811961755"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:09.286996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.286985 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:09.287202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.287003 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:09.287202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.287015 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:09.288645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.288621 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs" pod "538b0f02-cb5e-43b0-bad3-380811961755" (UID: "538b0f02-cb5e-43b0-bad3-380811961755"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:09.288770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.288654 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm" pod "538b0f02-cb5e-43b0-bad3-380811961755" (UID: "538b0f02-cb5e-43b0-bad3-380811961755"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:09.288770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.288687 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2" pod "538b0f02-cb5e-43b0-bad3-380811961755" (UID: "538b0f02-cb5e-43b0-bad3-380811961755"). InnerVolumeSpecName "kube-api-access-th8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:09.387903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.387867 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b0f02-cb5e-43b0-bad3-380811961755-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:09.387903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.387899 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b0f02-cb5e-43b0-bad3-380811961755-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:09.388104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.387915 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th8v2\" (UniqueName: \"kubernetes.io/projected/538b0f02-cb5e-43b0-bad3-380811961755-kube-api-access-th8v2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:09.891918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.891875 5647 generic.go:356] "Generic (PLEG): container finished" podID="538b0f02-cb5e-43b0-bad3-380811961755" containerID="fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d" exitCode=0 Nov 02 17:42:09.892109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.891931 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" Nov 02 17:42:09.892109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.891960 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" event={"ID":"538b0f02-cb5e-43b0-bad3-380811961755","Type":"ContainerDied","Data":"fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d"} Nov 02 17:42:09.892109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.892018 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f" event={"ID":"538b0f02-cb5e-43b0-bad3-380811961755","Type":"ContainerDied","Data":"0b6b0cfceebc54cc1a06952eb1bdc29511521d53870a744a7bd8999ab6e3e1b9"} Nov 02 17:42:09.892109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.892034 5647 scope.go:122] "RemoveContainer" containerID="fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d" Nov 02 17:42:09.901987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.901961 5647 scope.go:122] "RemoveContainer" containerID="fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d" Nov 02 17:42:09.902266 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:42:09.902243 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d\": container with ID starting with fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d not found: ID does not exist" containerID="fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d" Nov 02 17:42:09.902328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:09.902276 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d"} err="failed to get container status \"fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d\": rpc error: code = NotFound desc = could not find container \"fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d\": container with ID starting with fe27ed5c8793c3270edeeec59da6dc0634603cc22668741b78728b33c833f65d not found: ID does not exist" Nov 02 17:42:10.057305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:10.057270 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f"] Nov 02 17:42:10.108249 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:10.108211 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-b-562d66ee-kserve-5896db4976-scw7f"] Nov 02 17:42:11.080995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.080954 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538b0f02-cb5e-43b0-bad3-380811961755" path="/var/lib/kubelet/pods/538b0f02-cb5e-43b0-bad3-380811961755/volumes" Nov 02 17:42:11.744559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.744525 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt"] Nov 02 17:42:11.745144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.745114 5647 cpu_manager.go:414] "containerMap: removing container" podUID="538b0f02-cb5e-43b0-bad3-380811961755" containerName="main" Nov 02 17:42:11.745144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.745136 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538b0f02-cb5e-43b0-bad3-380811961755" containerName="main" Nov 02 17:42:11.745345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.745331 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="538b0f02-cb5e-43b0-bad3-380811961755" containerName="main" Nov 02 17:42:11.750239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.750218 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.804993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.804943 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt"] Nov 02 17:42:11.814540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.814513 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.814697 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.814559 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.814765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.814683 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2fn\" (UniqueName: \"kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.852746 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.852710 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:42:11.852916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.852710 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:42:11.916423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.916389 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2fn\" (UniqueName: \"kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.916608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.916512 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.916608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.916548 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.917362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.917295 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.919119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.919095 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:11.974131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:11.974100 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2fn\" (UniqueName: \"kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn\") pod \"maas-api-revoke-sub-fe8f212b-782bt\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:12.063426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:12.063348 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:12.435975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:12.435946 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt"] Nov 02 17:42:12.437786 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:12.437754 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb72637d_f6a7_4c8b_8adc_2c2952835878.slice/crio-d82f5a6625f5c35e5269aaa83df761ec0801224494889fd4d381a15e6d16c37e WatchSource:0}: Error finding container d82f5a6625f5c35e5269aaa83df761ec0801224494889fd4d381a15e6d16c37e: Status 404 returned error can't find the container with id d82f5a6625f5c35e5269aaa83df761ec0801224494889fd4d381a15e6d16c37e Nov 02 17:42:12.913109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:12.913064 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" event={"ID":"fb72637d-f6a7-4c8b-8adc-2c2952835878","Type":"ContainerStarted","Data":"7831274d9902b0db79421ed72505f5e719ed9c47f4807fbcc282090a83393b75"} Nov 02 17:42:12.913290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:12.913174 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" event={"ID":"fb72637d-f6a7-4c8b-8adc-2c2952835878","Type":"ContainerStarted","Data":"d82f5a6625f5c35e5269aaa83df761ec0801224494889fd4d381a15e6d16c37e"} Nov 02 17:42:13.029241 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:13.029146 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" podStartSLOduration=2.029133788 podStartE2EDuration="2.029133788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:13.028306339 +0000 UTC m=+1120.101833543" watchObservedRunningTime="2026-11-02 17:42:13.029133788 +0000 UTC m=+1120.102660992" Nov 02 17:42:13.919931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:13.919891 5647 generic.go:356] "Generic (PLEG): container finished" podID="fb72637d-f6a7-4c8b-8adc-2c2952835878" containerID="7831274d9902b0db79421ed72505f5e719ed9c47f4807fbcc282090a83393b75" exitCode=0 Nov 02 17:42:13.920357 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:13.919978 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" event={"ID":"fb72637d-f6a7-4c8b-8adc-2c2952835878","Type":"ContainerDied","Data":"7831274d9902b0db79421ed72505f5e719ed9c47f4807fbcc282090a83393b75"} Nov 02 17:42:15.077690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.077662 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:15.150984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.150949 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca\") pod \"fb72637d-f6a7-4c8b-8adc-2c2952835878\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " Nov 02 17:42:15.151194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.151012 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn\" (UniqueName: \"kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn\") pod \"fb72637d-f6a7-4c8b-8adc-2c2952835878\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " Nov 02 17:42:15.151194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.151156 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca\") pod \"fb72637d-f6a7-4c8b-8adc-2c2952835878\" (UID: \"fb72637d-f6a7-4c8b-8adc-2c2952835878\") " Nov 02 17:42:15.151523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.151499 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca" pod "fb72637d-f6a7-4c8b-8adc-2c2952835878" (UID: "fb72637d-f6a7-4c8b-8adc-2c2952835878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:42:15.153360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.153336 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn" pod "fb72637d-f6a7-4c8b-8adc-2c2952835878" (UID: "fb72637d-f6a7-4c8b-8adc-2c2952835878"). InnerVolumeSpecName "kube-api-access-cc2fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:15.153505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.153490 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca" pod "fb72637d-f6a7-4c8b-8adc-2c2952835878" (UID: "fb72637d-f6a7-4c8b-8adc-2c2952835878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:15.252952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.252864 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:15.252952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.252906 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc2fn\" (UniqueName: \"kubernetes.io/projected/fb72637d-f6a7-4c8b-8adc-2c2952835878-kube-api-access-cc2fn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:15.252952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.252921 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb72637d-f6a7-4c8b-8adc-2c2952835878-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:15.933476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.933443 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" Nov 02 17:42:15.933652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.933480 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt" event={"ID":"fb72637d-f6a7-4c8b-8adc-2c2952835878","Type":"ContainerDied","Data":"d82f5a6625f5c35e5269aaa83df761ec0801224494889fd4d381a15e6d16c37e"} Nov 02 17:42:15.933652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:15.933529 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d82f5a6625f5c35e5269aaa83df761ec0801224494889fd4d381a15e6d16c37e" Nov 02 17:42:20.625178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.625139 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" podUID="d41c0d6b-7bdf-48ea-bf44-860854190ae3" containerName="main" containerID="cri-o://920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb" gracePeriod=45 Nov 02 17:42:20.886116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.886053 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:42:20.962992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.962958 5647 generic.go:356] "Generic (PLEG): container finished" podID="d41c0d6b-7bdf-48ea-bf44-860854190ae3" containerID="920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb" exitCode=0 Nov 02 17:42:20.963229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.963001 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" Nov 02 17:42:20.963229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.963051 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" event={"ID":"d41c0d6b-7bdf-48ea-bf44-860854190ae3","Type":"ContainerDied","Data":"920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb"} Nov 02 17:42:20.963229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.963103 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6" event={"ID":"d41c0d6b-7bdf-48ea-bf44-860854190ae3","Type":"ContainerDied","Data":"d424cf93fff316f5088f12086c344a497897ecc461aaa4d97d93bc006998d88f"} Nov 02 17:42:20.963229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.963125 5647 scope.go:122] "RemoveContainer" containerID="920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb" Nov 02 17:42:20.972601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.972583 5647 scope.go:122] "RemoveContainer" containerID="920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb" Nov 02 17:42:20.972924 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:42:20.972903 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb\": container with ID starting with 920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb not found: ID does not exist" containerID="920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb" Nov 02 17:42:20.972997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:20.972932 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb"} err="failed to get container status \"920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb\": rpc error: code = NotFound desc = could not find container \"920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb\": container with ID starting with 920f90f7b0eb8eea3a09172b61255b5897cddc377616a2fdd6bfa0dcd3323afb not found: ID does not exist" Nov 02 17:42:21.021592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021550 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home\") pod \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " Nov 02 17:42:21.021803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021617 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm\") pod \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " Nov 02 17:42:21.021803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021639 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs\") pod \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " Nov 02 17:42:21.021803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021675 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir\") pod \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " Nov 02 17:42:21.021803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021724 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache\") pod \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " Nov 02 17:42:21.022011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021881 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn\" (UniqueName: \"kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn\") pod \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\" (UID: \"d41c0d6b-7bdf-48ea-bf44-860854190ae3\") " Nov 02 17:42:21.022011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.021909 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home" pod "d41c0d6b-7bdf-48ea-bf44-860854190ae3" (UID: "d41c0d6b-7bdf-48ea-bf44-860854190ae3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:21.022121 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.022039 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir" pod "d41c0d6b-7bdf-48ea-bf44-860854190ae3" (UID: "d41c0d6b-7bdf-48ea-bf44-860854190ae3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:21.022121 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.022080 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache" pod "d41c0d6b-7bdf-48ea-bf44-860854190ae3" (UID: "d41c0d6b-7bdf-48ea-bf44-860854190ae3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:21.022368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.022349 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:21.022445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.022372 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:21.022445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.022386 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:21.024098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.024075 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs" pod "d41c0d6b-7bdf-48ea-bf44-860854190ae3" (UID: "d41c0d6b-7bdf-48ea-bf44-860854190ae3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:21.024201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.024120 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn" pod "d41c0d6b-7bdf-48ea-bf44-860854190ae3" (UID: "d41c0d6b-7bdf-48ea-bf44-860854190ae3"). InnerVolumeSpecName "kube-api-access-42nhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:21.024314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.024298 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm" pod "d41c0d6b-7bdf-48ea-bf44-860854190ae3" (UID: "d41c0d6b-7bdf-48ea-bf44-860854190ae3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:42:21.123340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.123303 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42nhn\" (UniqueName: \"kubernetes.io/projected/d41c0d6b-7bdf-48ea-bf44-860854190ae3-kube-api-access-42nhn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:21.123340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.123335 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41c0d6b-7bdf-48ea-bf44-860854190ae3-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:21.123340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.123346 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41c0d6b-7bdf-48ea-bf44-860854190ae3-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:21.378287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.378256 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6"] Nov 02 17:42:21.385208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:21.385184 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/e2e-conflict-a-562d66ee-kserve-8f4db5c9d-zwzz6"] Nov 02 17:42:23.081004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:23.080970 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41c0d6b-7bdf-48ea-bf44-860854190ae3" path="/var/lib/kubelet/pods/d41c0d6b-7bdf-48ea-bf44-860854190ae3/volumes" Nov 02 17:42:24.131665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:24.131632 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:42:27.864960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.864927 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6"] Nov 02 17:42:27.865467 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.865452 5647 cpu_manager.go:414] "containerMap: removing container" podUID="fb72637d-f6a7-4c8b-8adc-2c2952835878" containerName="revoke-keys" Nov 02 17:42:27.865510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.865471 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb72637d-f6a7-4c8b-8adc-2c2952835878" containerName="revoke-keys" Nov 02 17:42:27.865510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.865501 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d41c0d6b-7bdf-48ea-bf44-860854190ae3" containerName="main" Nov 02 17:42:27.865510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.865508 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41c0d6b-7bdf-48ea-bf44-860854190ae3" containerName="main" Nov 02 17:42:27.865607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.865591 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41c0d6b-7bdf-48ea-bf44-860854190ae3" containerName="main" Nov 02 17:42:27.865644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.865607 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb72637d-f6a7-4c8b-8adc-2c2952835878" containerName="revoke-keys" Nov 02 17:42:27.870314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.870293 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:27.924991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.924953 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6"] Nov 02 17:42:27.972393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.972357 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:42:27.972393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:27.972392 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:42:28.007873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.007844 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.008055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.007905 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.008132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.008109 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4r69\" (UniqueName: \"kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.108852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.108806 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.109091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.108967 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4r69\" (UniqueName: \"kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.109091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.109005 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.109591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.109563 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.111531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.111508 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.166539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.166458 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4r69\" (UniqueName: \"kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69\") pod \"maas-api-revoke-sub-6cbe9572-bhxp6\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.183312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.183284 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:28.557419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:28.557390 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6"] Nov 02 17:42:28.559087 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:28.559063 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32877e8_d26f_48e5_975d_3c3c222c85bd.slice/crio-2fe47b0045da739d9397e9422012c0d19768a739667394cadc231c8b785549b3 WatchSource:0}: Error finding container 2fe47b0045da739d9397e9422012c0d19768a739667394cadc231c8b785549b3: Status 404 returned error can't find the container with id 2fe47b0045da739d9397e9422012c0d19768a739667394cadc231c8b785549b3 Nov 02 17:42:29.093126 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.093086 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" event={"ID":"e32877e8-d26f-48e5-975d-3c3c222c85bd","Type":"ContainerStarted","Data":"0aeea195a2c4ccd847c155137f9cf15a4f45b7231244298dc1a66aaf79aa2e68"} Nov 02 17:42:29.093604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.093140 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" event={"ID":"e32877e8-d26f-48e5-975d-3c3c222c85bd","Type":"ContainerStarted","Data":"2fe47b0045da739d9397e9422012c0d19768a739667394cadc231c8b785549b3"} Nov 02 17:42:29.207955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.207905 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" podStartSLOduration=2.207890059 podStartE2EDuration="2.207890059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:29.207040509 +0000 UTC m=+1136.280567711" watchObservedRunningTime="2026-11-02 17:42:29.207890059 +0000 UTC m=+1136.281417298" Nov 02 17:42:29.753034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.752952 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz"] Nov 02 17:42:29.756702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.756685 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.812805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.812770 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz"] Nov 02 17:42:29.830074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.830040 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lj6r\" (UniqueName: \"kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.830260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.830121 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.830317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.830257 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.931265 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.931213 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.931435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.931285 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.931435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.931361 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj6r\" (UniqueName: \"kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.932087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.932060 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.933845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.933826 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:29.991420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:29.991371 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj6r\" (UniqueName: \"kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r\") pod \"maas-api-revoke-sub-e1160493-qf4jz\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:30.069950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.069866 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:30.101110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.101077 5647 generic.go:356] "Generic (PLEG): container finished" podID="e32877e8-d26f-48e5-975d-3c3c222c85bd" containerID="0aeea195a2c4ccd847c155137f9cf15a4f45b7231244298dc1a66aaf79aa2e68" exitCode=0 Nov 02 17:42:30.101574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.101165 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" event={"ID":"e32877e8-d26f-48e5-975d-3c3c222c85bd","Type":"ContainerDied","Data":"0aeea195a2c4ccd847c155137f9cf15a4f45b7231244298dc1a66aaf79aa2e68"} Nov 02 17:42:30.341315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.341283 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb"] Nov 02 17:42:30.346303 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.346280 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.404174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.404145 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb"] Nov 02 17:42:30.436900 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.436860 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.436916 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.436933 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.437070 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.437123 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.437189 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437351 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.437283 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdhj\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437413 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.437354 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.437502 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.437425 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.454994 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.454942 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz"] Nov 02 17:42:30.539055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539022 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdhj\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539058 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539098 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539160 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539189 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539255 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539382 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539402 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539586 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539693 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539851 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539875 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.539955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.539881 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.602505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.602432 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.602888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.602863 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.602971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.602871 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.603097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.603076 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdhj\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:30.660584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:30.660557 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:31.089389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.089365 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb"] Nov 02 17:42:31.090886 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:31.090862 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0349f5cd_00ad_4b9e_9be8_587f1ae34f60.slice/crio-d999237c6a7f0f88584cbff9c80bc74e3c3ac63c18bdd62339a6275a5fa39b94 WatchSource:0}: Error finding container d999237c6a7f0f88584cbff9c80bc74e3c3ac63c18bdd62339a6275a5fa39b94: Status 404 returned error can't find the container with id d999237c6a7f0f88584cbff9c80bc74e3c3ac63c18bdd62339a6275a5fa39b94 Nov 02 17:42:31.093325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.093286 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:42:31.093427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.093369 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:42:31.093427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.093413 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:42:31.108872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.108840 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" event={"ID":"3ed45447-9b90-42b6-ab6a-aa3b41153882","Type":"ContainerStarted","Data":"eda40cf6b1b845297b46d252b05df2378a35ce926920dc4856970daff9fe2471"} Nov 02 17:42:31.109305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.108894 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" event={"ID":"3ed45447-9b90-42b6-ab6a-aa3b41153882","Type":"ContainerStarted","Data":"44bc845e8ece6e994ac3c12ac5e2967093831c91f20d9bdc55ddd24d8ce24050"} Nov 02 17:42:31.110151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.110127 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" event={"ID":"0349f5cd-00ad-4b9e-9be8-587f1ae34f60","Type":"ContainerStarted","Data":"d999237c6a7f0f88584cbff9c80bc74e3c3ac63c18bdd62339a6275a5fa39b94"} Nov 02 17:42:31.229329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.229250 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" podStartSLOduration=2.22922858 podStartE2EDuration="2.22922858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:31.225433762 +0000 UTC m=+1138.298960976" watchObservedRunningTime="2026-11-02 17:42:31.22922858 +0000 UTC m=+1138.302755778" Nov 02 17:42:31.286498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.286470 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:31.449135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.449101 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69\" (UniqueName: \"kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69\") pod \"e32877e8-d26f-48e5-975d-3c3c222c85bd\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " Nov 02 17:42:31.449332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.449216 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca\") pod \"e32877e8-d26f-48e5-975d-3c3c222c85bd\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " Nov 02 17:42:31.449401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.449357 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca\") pod \"e32877e8-d26f-48e5-975d-3c3c222c85bd\" (UID: \"e32877e8-d26f-48e5-975d-3c3c222c85bd\") " Nov 02 17:42:31.449765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.449722 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca" pod "e32877e8-d26f-48e5-975d-3c3c222c85bd" (UID: "e32877e8-d26f-48e5-975d-3c3c222c85bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:42:31.449921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.449898 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:31.451965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.451943 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca" pod "e32877e8-d26f-48e5-975d-3c3c222c85bd" (UID: "e32877e8-d26f-48e5-975d-3c3c222c85bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:31.452157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.452129 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69" pod "e32877e8-d26f-48e5-975d-3c3c222c85bd" (UID: "e32877e8-d26f-48e5-975d-3c3c222c85bd"). InnerVolumeSpecName "kube-api-access-c4r69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:31.550872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.550831 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4r69\" (UniqueName: \"kubernetes.io/projected/e32877e8-d26f-48e5-975d-3c3c222c85bd-kube-api-access-c4r69\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:31.550872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:31.550867 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32877e8-d26f-48e5-975d-3c3c222c85bd-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:32.134630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.134597 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" Nov 02 17:42:32.134630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.134621 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6" event={"ID":"e32877e8-d26f-48e5-975d-3c3c222c85bd","Type":"ContainerDied","Data":"2fe47b0045da739d9397e9422012c0d19768a739667394cadc231c8b785549b3"} Nov 02 17:42:32.135197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.134678 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fe47b0045da739d9397e9422012c0d19768a739667394cadc231c8b785549b3" Nov 02 17:42:32.136411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.136386 5647 generic.go:356] "Generic (PLEG): container finished" podID="3ed45447-9b90-42b6-ab6a-aa3b41153882" containerID="eda40cf6b1b845297b46d252b05df2378a35ce926920dc4856970daff9fe2471" exitCode=0 Nov 02 17:42:32.136555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.136474 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" event={"ID":"3ed45447-9b90-42b6-ab6a-aa3b41153882","Type":"ContainerDied","Data":"eda40cf6b1b845297b46d252b05df2378a35ce926920dc4856970daff9fe2471"} Nov 02 17:42:32.138102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.138077 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" event={"ID":"0349f5cd-00ad-4b9e-9be8-587f1ae34f60","Type":"ContainerStarted","Data":"e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e"} Nov 02 17:42:32.468893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.468782 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podStartSLOduration=2.468766372 podStartE2EDuration="2.468766372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:32.466416714 +0000 UTC m=+1139.539943918" watchObservedRunningTime="2026-11-02 17:42:32.468766372 +0000 UTC m=+1139.542293646" Nov 02 17:42:32.661211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.661177 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:32.662808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.662783 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 17:42:32.662950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:32.662835 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 17:42:33.329182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.329151 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:33.370609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.370585 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca\") pod \"3ed45447-9b90-42b6-ab6a-aa3b41153882\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " Nov 02 17:42:33.370714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.370637 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r\" (UniqueName: \"kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r\") pod \"3ed45447-9b90-42b6-ab6a-aa3b41153882\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " Nov 02 17:42:33.370822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.370807 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca\") pod \"3ed45447-9b90-42b6-ab6a-aa3b41153882\" (UID: \"3ed45447-9b90-42b6-ab6a-aa3b41153882\") " Nov 02 17:42:33.371199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.371174 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca" pod "3ed45447-9b90-42b6-ab6a-aa3b41153882" (UID: "3ed45447-9b90-42b6-ab6a-aa3b41153882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:42:33.371303 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.371286 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:33.372854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.372830 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r" pod "3ed45447-9b90-42b6-ab6a-aa3b41153882" (UID: "3ed45447-9b90-42b6-ab6a-aa3b41153882"). InnerVolumeSpecName "kube-api-access-4lj6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:33.372854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.372844 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca" pod "3ed45447-9b90-42b6-ab6a-aa3b41153882" (UID: "3ed45447-9b90-42b6-ab6a-aa3b41153882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:33.472329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.472255 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed45447-9b90-42b6-ab6a-aa3b41153882-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:33.472329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.472282 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lj6r\" (UniqueName: \"kubernetes.io/projected/3ed45447-9b90-42b6-ab6a-aa3b41153882-kube-api-access-4lj6r\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:33.662169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.662127 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 17:42:33.662372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:33.662187 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 17:42:34.150446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:34.150403 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" event={"ID":"3ed45447-9b90-42b6-ab6a-aa3b41153882","Type":"ContainerDied","Data":"44bc845e8ece6e994ac3c12ac5e2967093831c91f20d9bdc55ddd24d8ce24050"} Nov 02 17:42:34.150615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:34.150460 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz" Nov 02 17:42:34.150615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:34.150469 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44bc845e8ece6e994ac3c12ac5e2967093831c91f20d9bdc55ddd24d8ce24050" Nov 02 17:42:34.662035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:34.661989 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 17:42:34.662442 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:34.662050 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 17:42:35.661477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:35.661439 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 17:42:35.661676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:35.661491 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 17:42:35.953946 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:42:35.953859 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:42:36.661451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:36.661410 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 17:42:36.661653 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:36.661465 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 17:42:37.661635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:37.661603 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 02 17:42:37.662072 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:37.661659 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 02 17:42:38.665873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:38.665844 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:38.666281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:38.666172 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:38.666821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:38.666796 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:42:39.052417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:39.052333 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg"] Nov 02 17:42:39.055333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:39.055309 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887cc672-2smgg"] Nov 02 17:42:39.080995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:39.080963 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c338d6-41aa-4e13-83b7-08c210f2f3b4" path="/var/lib/kubelet/pods/08c338d6-41aa-4e13-83b7-08c210f2f3b4/volumes" Nov 02 17:42:40.498034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498002 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9"] Nov 02 17:42:40.498476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498463 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e32877e8-d26f-48e5-975d-3c3c222c85bd" containerName="revoke-keys" Nov 02 17:42:40.498531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498477 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32877e8-d26f-48e5-975d-3c3c222c85bd" containerName="revoke-keys" Nov 02 17:42:40.498531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498492 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3ed45447-9b90-42b6-ab6a-aa3b41153882" containerName="revoke-keys" Nov 02 17:42:40.498531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498497 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed45447-9b90-42b6-ab6a-aa3b41153882" containerName="revoke-keys" Nov 02 17:42:40.498622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498585 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed45447-9b90-42b6-ab6a-aa3b41153882" containerName="revoke-keys" Nov 02 17:42:40.498622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.498601 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32877e8-d26f-48e5-975d-3c3c222c85bd" containerName="revoke-keys" Nov 02 17:42:40.503216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.503195 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.557559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.557531 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9"] Nov 02 17:42:40.605386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.605355 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:42:40.605547 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.605439 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:42:40.681937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.681896 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.682124 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.681947 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.682124 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.682055 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9kvh\" (UniqueName: \"kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.783769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.783661 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.783769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.783705 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.783978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.783783 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9kvh\" (UniqueName: \"kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.784799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.784768 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.787212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.787181 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:40.841918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:40.841885 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9kvh\" (UniqueName: \"kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh\") pod \"maas-api-revoke-sub-c21765ce-g5fx9\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:41.117462 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:41.117426 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:41.492483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:41.492452 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9"] Nov 02 17:42:41.493984 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:41.493959 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53432b2_16cb_4a66_a4d8_c24101062f1b.slice/crio-1a51d23df6c381fac8e2075d170eccf5df62bcb5c1a1e8526b7e764aac527cbf WatchSource:0}: Error finding container 1a51d23df6c381fac8e2075d170eccf5df62bcb5c1a1e8526b7e764aac527cbf: Status 404 returned error can't find the container with id 1a51d23df6c381fac8e2075d170eccf5df62bcb5c1a1e8526b7e764aac527cbf Nov 02 17:42:42.202410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:42.202369 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" event={"ID":"c53432b2-16cb-4a66-a4d8-c24101062f1b","Type":"ContainerStarted","Data":"e5e22328f987b2f97a24570c8454e25aee4ce561c9ec35630bf3e7a20d4dbb39"} Nov 02 17:42:42.202926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:42.202435 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" event={"ID":"c53432b2-16cb-4a66-a4d8-c24101062f1b","Type":"ContainerStarted","Data":"1a51d23df6c381fac8e2075d170eccf5df62bcb5c1a1e8526b7e764aac527cbf"} Nov 02 17:42:42.318683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:42.318634 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" podStartSLOduration=2.318619759 podStartE2EDuration="2.318619759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:42.318453994 +0000 UTC m=+1149.391981198" watchObservedRunningTime="2026-11-02 17:42:42.318619759 +0000 UTC m=+1149.392146963" Nov 02 17:42:43.211630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.211595 5647 generic.go:356] "Generic (PLEG): container finished" podID="c53432b2-16cb-4a66-a4d8-c24101062f1b" containerID="e5e22328f987b2f97a24570c8454e25aee4ce561c9ec35630bf3e7a20d4dbb39" exitCode=0 Nov 02 17:42:43.212201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.211692 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" event={"ID":"c53432b2-16cb-4a66-a4d8-c24101062f1b","Type":"ContainerDied","Data":"e5e22328f987b2f97a24570c8454e25aee4ce561c9ec35630bf3e7a20d4dbb39"} Nov 02 17:42:43.909852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.909819 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn"] Nov 02 17:42:43.913831 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.913810 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:43.943338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.943303 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54mls\" (UniqueName: \"kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:43.943516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.943358 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:43.943516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.943504 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:43.970997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:43.970965 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn"] Nov 02 17:42:44.044953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.044920 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.045162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.044985 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54mls\" (UniqueName: \"kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.045162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.045024 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.045654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.045633 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.047368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.047350 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.104008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.103981 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54mls\" (UniqueName: \"kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls\") pod \"maas-api-revoke-sub-14d9bf31-gsktn\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.227364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.227328 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:44.369098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.369068 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:44.449890 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.449860 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca\") pod \"c53432b2-16cb-4a66-a4d8-c24101062f1b\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " Nov 02 17:42:44.450088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.449987 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh\" (UniqueName: \"kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh\") pod \"c53432b2-16cb-4a66-a4d8-c24101062f1b\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " Nov 02 17:42:44.450088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.450041 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca\") pod \"c53432b2-16cb-4a66-a4d8-c24101062f1b\" (UID: \"c53432b2-16cb-4a66-a4d8-c24101062f1b\") " Nov 02 17:42:44.450439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.450416 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca" pod "c53432b2-16cb-4a66-a4d8-c24101062f1b" (UID: "c53432b2-16cb-4a66-a4d8-c24101062f1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:42:44.452092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.452066 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca" pod "c53432b2-16cb-4a66-a4d8-c24101062f1b" (UID: "c53432b2-16cb-4a66-a4d8-c24101062f1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:44.452203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.452162 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh" pod "c53432b2-16cb-4a66-a4d8-c24101062f1b" (UID: "c53432b2-16cb-4a66-a4d8-c24101062f1b"). InnerVolumeSpecName "kube-api-access-k9kvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:44.551227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.551142 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:44.551227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.551174 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c53432b2-16cb-4a66-a4d8-c24101062f1b-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:44.551227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.551185 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9kvh\" (UniqueName: \"kubernetes.io/projected/c53432b2-16cb-4a66-a4d8-c24101062f1b-kube-api-access-k9kvh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:44.614447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:44.613102 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn"] Nov 02 17:42:45.225028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:45.224997 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" Nov 02 17:42:45.225227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:45.225037 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9" event={"ID":"c53432b2-16cb-4a66-a4d8-c24101062f1b","Type":"ContainerDied","Data":"1a51d23df6c381fac8e2075d170eccf5df62bcb5c1a1e8526b7e764aac527cbf"} Nov 02 17:42:45.225227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:45.225096 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a51d23df6c381fac8e2075d170eccf5df62bcb5c1a1e8526b7e764aac527cbf" Nov 02 17:42:45.226928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:45.226899 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" event={"ID":"19e5fb23-0d06-4ff0-a202-072325c055f0","Type":"ContainerStarted","Data":"379d494c437f78c4aaa294f427a239d6779ce0bb5d11e0f593707359c4a42d61"} Nov 02 17:42:45.227071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:45.226949 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" event={"ID":"19e5fb23-0d06-4ff0-a202-072325c055f0","Type":"ContainerStarted","Data":"c2c0581b0dd4ce6f436507224043c93c108b2808b0fe7b3b4850e24311dd1667"} Nov 02 17:42:45.342192 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:45.342096 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" podStartSLOduration=2.342081724 podStartE2EDuration="2.342081724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:45.341335087 +0000 UTC m=+1152.414862292" watchObservedRunningTime="2026-11-02 17:42:45.342081724 +0000 UTC m=+1152.415608972" Nov 02 17:42:46.234622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:46.234588 5647 generic.go:356] "Generic (PLEG): container finished" podID="19e5fb23-0d06-4ff0-a202-072325c055f0" containerID="379d494c437f78c4aaa294f427a239d6779ce0bb5d11e0f593707359c4a42d61" exitCode=0 Nov 02 17:42:46.234821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:46.234655 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" event={"ID":"19e5fb23-0d06-4ff0-a202-072325c055f0","Type":"ContainerDied","Data":"379d494c437f78c4aaa294f427a239d6779ce0bb5d11e0f593707359c4a42d61"} Nov 02 17:42:47.402499 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.402470 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:47.489946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.489909 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca\") pod \"19e5fb23-0d06-4ff0-a202-072325c055f0\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " Nov 02 17:42:47.490117 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.489998 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca\") pod \"19e5fb23-0d06-4ff0-a202-072325c055f0\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " Nov 02 17:42:47.490117 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.490056 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls\" (UniqueName: \"kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls\") pod \"19e5fb23-0d06-4ff0-a202-072325c055f0\" (UID: \"19e5fb23-0d06-4ff0-a202-072325c055f0\") " Nov 02 17:42:47.490446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.490385 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca" pod "19e5fb23-0d06-4ff0-a202-072325c055f0" (UID: "19e5fb23-0d06-4ff0-a202-072325c055f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:42:47.492195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.492166 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca" pod "19e5fb23-0d06-4ff0-a202-072325c055f0" (UID: "19e5fb23-0d06-4ff0-a202-072325c055f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:47.492284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.492247 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls" pod "19e5fb23-0d06-4ff0-a202-072325c055f0" (UID: "19e5fb23-0d06-4ff0-a202-072325c055f0"). InnerVolumeSpecName "kube-api-access-54mls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:47.591799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.591764 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54mls\" (UniqueName: \"kubernetes.io/projected/19e5fb23-0d06-4ff0-a202-072325c055f0-kube-api-access-54mls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:47.591799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.591796 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:47.591978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:47.591811 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19e5fb23-0d06-4ff0-a202-072325c055f0-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:48.246760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:48.246701 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" Nov 02 17:42:48.246953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:48.246753 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn" event={"ID":"19e5fb23-0d06-4ff0-a202-072325c055f0","Type":"ContainerDied","Data":"c2c0581b0dd4ce6f436507224043c93c108b2808b0fe7b3b4850e24311dd1667"} Nov 02 17:42:48.246953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:48.246809 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c0581b0dd4ce6f436507224043c93c108b2808b0fe7b3b4850e24311dd1667" Nov 02 17:42:52.992525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:52.992492 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:42:55.268815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.268713 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g"] Nov 02 17:42:55.269248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.269234 5647 cpu_manager.go:414] "containerMap: removing container" podUID="19e5fb23-0d06-4ff0-a202-072325c055f0" containerName="revoke-keys" Nov 02 17:42:55.269290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.269249 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e5fb23-0d06-4ff0-a202-072325c055f0" containerName="revoke-keys" Nov 02 17:42:55.269290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.269263 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c53432b2-16cb-4a66-a4d8-c24101062f1b" containerName="revoke-keys" Nov 02 17:42:55.269290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.269268 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53432b2-16cb-4a66-a4d8-c24101062f1b" containerName="revoke-keys" Nov 02 17:42:55.269381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.269339 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e5fb23-0d06-4ff0-a202-072325c055f0" containerName="revoke-keys" Nov 02 17:42:55.269381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.269347 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53432b2-16cb-4a66-a4d8-c24101062f1b" containerName="revoke-keys" Nov 02 17:42:55.272816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.272799 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.328841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.328808 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g"] Nov 02 17:42:55.376695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.376660 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"kube-root-ca.crt\"" Nov 02 17:42:55.376695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.376686 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"openshift-service-ca.crt\"" Nov 02 17:42:55.376695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.376699 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"default-dockercfg-fvn5n\"" Nov 02 17:42:55.376695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.376689 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-6e3175\"/\"distinct-2-w3-6e3175-kserve-self-signed-certs\"" Nov 02 17:42:55.398698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.398666 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.398858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.398763 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpll\" (UniqueName: \"kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.398927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.398905 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.398968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.398950 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.399002 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.398975 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.399036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.399017 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.500759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.500247 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.500759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.500357 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpll\" (UniqueName: \"kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.500759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.500423 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.500759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.500455 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.500759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.500485 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.500759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.500532 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.501209 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.501041 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.501302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.501281 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.502951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.502049 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.506265 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.506211 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.507040 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.506986 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.563362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.563290 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpll\" (UniqueName: \"kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll\") pod \"distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:55.586026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:55.585994 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:56.030671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.030645 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g"] Nov 02 17:42:56.032251 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:56.032226 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20469e2c_c670_4c90_9102_6b4e3ecc8b7a.slice/crio-51218e383a3fd273dbb2fd478baef98a44eae4abb1eebc054d7b577233ed01d2 WatchSource:0}: Error finding container 51218e383a3fd273dbb2fd478baef98a44eae4abb1eebc054d7b577233ed01d2: Status 404 returned error can't find the container with id 51218e383a3fd273dbb2fd478baef98a44eae4abb1eebc054d7b577233ed01d2 Nov 02 17:42:56.298597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.298505 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" event={"ID":"20469e2c-c670-4c90-9102-6b4e3ecc8b7a","Type":"ContainerStarted","Data":"56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb"} Nov 02 17:42:56.298597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.298551 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" event={"ID":"20469e2c-c670-4c90-9102-6b4e3ecc8b7a","Type":"ContainerStarted","Data":"51218e383a3fd273dbb2fd478baef98a44eae4abb1eebc054d7b577233ed01d2"} Nov 02 17:42:56.414601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.414539 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podStartSLOduration=1.414527143 podStartE2EDuration="1.414527143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:56.413177842 +0000 UTC m=+1163.486705046" watchObservedRunningTime="2026-11-02 17:42:56.414527143 +0000 UTC m=+1163.488054346" Nov 02 17:42:56.608869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.586331 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:42:56.763956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.763919 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h"] Nov 02 17:42:56.767916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.767895 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.817712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.817681 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.817918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.817803 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vpp7\" (UniqueName: \"kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.817918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.817844 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.823329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.823299 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:42:56.823489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.823336 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:42:56.823666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.823646 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h"] Nov 02 17:42:56.918971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.918878 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.918971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.918946 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vpp7\" (UniqueName: \"kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.919228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.919027 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.919765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.919668 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.921529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.921500 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:56.978015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:56.977978 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vpp7\" (UniqueName: \"kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7\") pod \"maas-api-revoke-sub-7fbf48cb-2st6h\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:57.047318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.047280 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9"] Nov 02 17:42:57.055752 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.051554 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2124691-gzft9"] Nov 02 17:42:57.080444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.080406 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dd25fe-423b-45da-8929-c42b9c60f1c9" path="/var/lib/kubelet/pods/47dd25fe-423b-45da-8929-c42b9c60f1c9/volumes" Nov 02 17:42:57.082107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.082089 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:57.460954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.460924 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h"] Nov 02 17:42:57.462627 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:57.462598 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16510d45_1769_4918_9c36_be628783bfeb.slice/crio-3b2dad04dbab5a76ca339daaed9d7e716c335a99858b26e028498f0ba5581165 WatchSource:0}: Error finding container 3b2dad04dbab5a76ca339daaed9d7e716c335a99858b26e028498f0ba5581165: Status 404 returned error can't find the container with id 3b2dad04dbab5a76ca339daaed9d7e716c335a99858b26e028498f0ba5581165 Nov 02 17:42:57.894495 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.894462 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p"] Nov 02 17:42:57.898570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.898549 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:57.991973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:57.991889 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p"] Nov 02 17:42:58.033008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.032975 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033027 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033153 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033254 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033291 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n4tk\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033391 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033423 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033538 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.033605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.033592 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135142 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135100 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n4tk\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135156 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135268 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135488 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135575 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135552 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135614 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135636 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135671 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135850 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.135942 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.135929 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.136061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.136026 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.136171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.136145 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.136418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.136399 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.136467 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.136405 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.138320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.138291 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.138520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.138502 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.194702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.194666 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.194875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.194728 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n4tk\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.212353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.212319 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:42:58.313677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.313644 5647 generic.go:356] "Generic (PLEG): container finished" podID="16510d45-1769-4918-9c36-be628783bfeb" containerID="5bd1e7e14fb7dc8e8dd15204c60ca0807b4b4b04f845750d7e64989fb124ba5a" exitCode=0 Nov 02 17:42:58.313865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.313728 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" event={"ID":"16510d45-1769-4918-9c36-be628783bfeb","Type":"ContainerDied","Data":"5bd1e7e14fb7dc8e8dd15204c60ca0807b4b4b04f845750d7e64989fb124ba5a"} Nov 02 17:42:58.313865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.313822 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" event={"ID":"16510d45-1769-4918-9c36-be628783bfeb","Type":"ContainerStarted","Data":"3b2dad04dbab5a76ca339daaed9d7e716c335a99858b26e028498f0ba5581165"} Nov 02 17:42:58.553874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.553840 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m"] Nov 02 17:42:58.557865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.557838 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.613588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.613326 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"kube-root-ca.crt\"" Nov 02 17:42:58.613588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.613474 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"openshift-service-ca.crt\"" Nov 02 17:42:58.613588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.613517 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"distinct-w2-a099c3-kserve-self-signed-certs\"" Nov 02 17:42:58.613588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.613562 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"default-dockercfg-2jpz8\"" Nov 02 17:42:58.616426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.616400 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m"] Nov 02 17:42:58.638355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.638328 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p"] Nov 02 17:42:58.641780 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:42:58.641715 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829651da_23ec_4567_8b0d_58677008d0a3.slice/crio-f4cdabdea1d832d3ac9373fca8f99041faf13c786265b83c4f7d18c940299f52 WatchSource:0}: Error finding container f4cdabdea1d832d3ac9373fca8f99041faf13c786265b83c4f7d18c940299f52: Status 404 returned error can't find the container with id f4cdabdea1d832d3ac9373fca8f99041faf13c786265b83c4f7d18c940299f52 Nov 02 17:42:58.641909 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.641860 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.641909 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.641896 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.642137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.642117 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw5b\" (UniqueName: \"kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.642184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.642157 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.642234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.642219 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.642277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.642248 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.644269 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.644230 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:42:58.644353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.644322 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:42:58.644401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.644365 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.744578 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.744830 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw5b\" (UniqueName: \"kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.744868 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.744896 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.744922 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.745040 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.745499 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.748769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.745781 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.763758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.749970 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.763758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.750554 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.763758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.753858 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.804784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.804750 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw5b\" (UniqueName: \"kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b\") pod \"distinct-w2-a099c3-kserve-89c85d586-gkk8m\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:58.870854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:58.870817 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:42:59.330759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.330116 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" event={"ID":"829651da-23ec-4567-8b0d-58677008d0a3","Type":"ContainerStarted","Data":"25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810"} Nov 02 17:42:59.330759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.330184 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" event={"ID":"829651da-23ec-4567-8b0d-58677008d0a3","Type":"ContainerStarted","Data":"f4cdabdea1d832d3ac9373fca8f99041faf13c786265b83c4f7d18c940299f52"} Nov 02 17:42:59.330759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.330672 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m"] Nov 02 17:42:59.454411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.454342 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podStartSLOduration=2.4543211 podStartE2EDuration="2.4543211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:59.449648164 +0000 UTC m=+1166.523175380" watchObservedRunningTime="2026-11-02 17:42:59.4543211 +0000 UTC m=+1166.527848303" Nov 02 17:42:59.499120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.499093 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:42:59.561069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.561028 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca\") pod \"16510d45-1769-4918-9c36-be628783bfeb\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " Nov 02 17:42:59.561477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.561173 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7\" (UniqueName: \"kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7\") pod \"16510d45-1769-4918-9c36-be628783bfeb\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " Nov 02 17:42:59.561477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.561246 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca\") pod \"16510d45-1769-4918-9c36-be628783bfeb\" (UID: \"16510d45-1769-4918-9c36-be628783bfeb\") " Nov 02 17:42:59.561477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.561452 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca" pod "16510d45-1769-4918-9c36-be628783bfeb" (UID: "16510d45-1769-4918-9c36-be628783bfeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:42:59.561679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.561660 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16510d45-1769-4918-9c36-be628783bfeb-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:59.563482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.563451 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7" pod "16510d45-1769-4918-9c36-be628783bfeb" (UID: "16510d45-1769-4918-9c36-be628783bfeb"). InnerVolumeSpecName "kube-api-access-7vpp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:42:59.563625 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.563502 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca" pod "16510d45-1769-4918-9c36-be628783bfeb" (UID: "16510d45-1769-4918-9c36-be628783bfeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:42:59.662861 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.662824 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vpp7\" (UniqueName: \"kubernetes.io/projected/16510d45-1769-4918-9c36-be628783bfeb-kube-api-access-7vpp7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:42:59.662861 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:42:59.662854 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16510d45-1769-4918-9c36-be628783bfeb-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:00.227007 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.226970 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:43:00.228294 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.228264 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:43:00.228444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.228322 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:43:00.253571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.253539 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc"] Nov 02 17:43:00.254072 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.254059 5647 cpu_manager.go:414] "containerMap: removing container" podUID="16510d45-1769-4918-9c36-be628783bfeb" containerName="revoke-keys" Nov 02 17:43:00.254123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.254074 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16510d45-1769-4918-9c36-be628783bfeb" containerName="revoke-keys" Nov 02 17:43:00.254170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.254161 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="16510d45-1769-4918-9c36-be628783bfeb" containerName="revoke-keys" Nov 02 17:43:00.267355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.267329 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.327350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.327309 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc"] Nov 02 17:43:00.343067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.343022 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" event={"ID":"0a942f59-c314-41d8-aa5f-d46eae01060e","Type":"ContainerStarted","Data":"89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af"} Nov 02 17:43:00.343235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.343076 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" event={"ID":"0a942f59-c314-41d8-aa5f-d46eae01060e","Type":"ContainerStarted","Data":"a792bf0270100b28f3748315f4f3468f3327b2175c4596614ae8167014b1105c"} Nov 02 17:43:00.344683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.344662 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" Nov 02 17:43:00.344852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.344700 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h" event={"ID":"16510d45-1769-4918-9c36-be628783bfeb","Type":"ContainerDied","Data":"3b2dad04dbab5a76ca339daaed9d7e716c335a99858b26e028498f0ba5581165"} Nov 02 17:43:00.344852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.344775 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2dad04dbab5a76ca339daaed9d7e716c335a99858b26e028498f0ba5581165" Nov 02 17:43:00.370348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.370316 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqkz\" (UniqueName: \"kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.370522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.370456 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.370599 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.370558 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.472184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.472145 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqkz\" (UniqueName: \"kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.472461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.472437 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.472591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.472571 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.473202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.473173 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.475235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.475212 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.500873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.500784 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podStartSLOduration=2.5007700550000003 podStartE2EDuration="2.500770055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:00.500476797 +0000 UTC m=+1167.574004001" watchObservedRunningTime="2026-11-02 17:43:00.500770055 +0000 UTC m=+1167.574297258" Nov 02 17:43:00.530143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.530107 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqkz\" (UniqueName: \"kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz\") pod \"maas-api-revoke-sub-1638eb12-j8xwc\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.581502 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.581472 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:00.871260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:00.871222 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:43:01.060713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:01.060685 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc"] Nov 02 17:43:01.061815 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:01.061785 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13333e50_d39a_4fbe_90d9_c67ed0e21139.slice/crio-b11b514be386cb4e1aae1dbd1e59e656ad19cf803d3443d3e231798719dec404 WatchSource:0}: Error finding container b11b514be386cb4e1aae1dbd1e59e656ad19cf803d3443d3e231798719dec404: Status 404 returned error can't find the container with id b11b514be386cb4e1aae1dbd1e59e656ad19cf803d3443d3e231798719dec404 Nov 02 17:43:01.219760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:01.217204 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:43:01.219760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:01.217284 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:43:01.352699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:01.352669 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" event={"ID":"13333e50-d39a-4fbe-90d9-c67ed0e21139","Type":"ContainerStarted","Data":"4b6ad7943b2408623d295138f12e734cfb9ce2e94dad8d58aa674c22e43cda38"} Nov 02 17:43:01.352865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:01.352723 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" event={"ID":"13333e50-d39a-4fbe-90d9-c67ed0e21139","Type":"ContainerStarted","Data":"b11b514be386cb4e1aae1dbd1e59e656ad19cf803d3443d3e231798719dec404"} Nov 02 17:43:02.227285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:02.227243 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:43:02.227981 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:02.227342 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:43:02.360564 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:02.360472 5647 generic.go:356] "Generic (PLEG): container finished" podID="13333e50-d39a-4fbe-90d9-c67ed0e21139" containerID="4b6ad7943b2408623d295138f12e734cfb9ce2e94dad8d58aa674c22e43cda38" exitCode=0 Nov 02 17:43:02.360564 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:02.360536 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" event={"ID":"13333e50-d39a-4fbe-90d9-c67ed0e21139","Type":"ContainerDied","Data":"4b6ad7943b2408623d295138f12e734cfb9ce2e94dad8d58aa674c22e43cda38"} Nov 02 17:43:03.222761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.222199 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:43:03.222761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.222263 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:43:03.599965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.599930 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:03.754634 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.754599 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca\") pod \"13333e50-d39a-4fbe-90d9-c67ed0e21139\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " Nov 02 17:43:03.755316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.755294 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca\") pod \"13333e50-d39a-4fbe-90d9-c67ed0e21139\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " Nov 02 17:43:03.755951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.755933 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz\" (UniqueName: \"kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz\") pod \"13333e50-d39a-4fbe-90d9-c67ed0e21139\" (UID: \"13333e50-d39a-4fbe-90d9-c67ed0e21139\") " Nov 02 17:43:03.756966 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.755780 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca" pod "13333e50-d39a-4fbe-90d9-c67ed0e21139" (UID: "13333e50-d39a-4fbe-90d9-c67ed0e21139"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:03.784390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.784350 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca" pod "13333e50-d39a-4fbe-90d9-c67ed0e21139" (UID: "13333e50-d39a-4fbe-90d9-c67ed0e21139"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:03.784831 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.784793 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz" pod "13333e50-d39a-4fbe-90d9-c67ed0e21139" (UID: "13333e50-d39a-4fbe-90d9-c67ed0e21139"). InnerVolumeSpecName "kube-api-access-fsqkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:03.862760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.862637 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqkz\" (UniqueName: \"kubernetes.io/projected/13333e50-d39a-4fbe-90d9-c67ed0e21139-kube-api-access-fsqkz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:03.862760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.862680 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:03.862760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:03.862695 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13333e50-d39a-4fbe-90d9-c67ed0e21139-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:04.218822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:04.216180 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:43:04.218822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:04.216244 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:43:04.375424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:04.375386 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" event={"ID":"13333e50-d39a-4fbe-90d9-c67ed0e21139","Type":"ContainerDied","Data":"b11b514be386cb4e1aae1dbd1e59e656ad19cf803d3443d3e231798719dec404"} Nov 02 17:43:04.375639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:04.375436 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b11b514be386cb4e1aae1dbd1e59e656ad19cf803d3443d3e231798719dec404" Nov 02 17:43:04.375639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:04.375449 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc" Nov 02 17:43:05.213140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:05.213095 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Nov 02 17:43:05.213637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:05.213169 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Nov 02 17:43:05.586484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:05.586457 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:43:05.600085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:05.600055 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:43:05.951725 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:05.951624 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:43:06.453135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:06.453103 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:43:07.253905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:07.253870 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:43:07.254098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:07.253927 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:07.612844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:07.612810 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:43:07.613326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:07.613218 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:43:07.613951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:07.613925 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:43:08.871524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:08.871488 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:43:08.884303 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:08.884277 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:43:09.471373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:09.471343 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:43:11.026103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.026072 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k"] Nov 02 17:43:11.026591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.026578 5647 cpu_manager.go:414] "containerMap: removing container" podUID="13333e50-d39a-4fbe-90d9-c67ed0e21139" containerName="revoke-keys" Nov 02 17:43:11.026642 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.026593 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13333e50-d39a-4fbe-90d9-c67ed0e21139" containerName="revoke-keys" Nov 02 17:43:11.026700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.026690 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="13333e50-d39a-4fbe-90d9-c67ed0e21139" containerName="revoke-keys" Nov 02 17:43:11.030282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.030265 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.086807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.086779 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k"] Nov 02 17:43:11.134455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.134414 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:11.134455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.134423 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:43:11.155844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.155812 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.156010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.155875 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.156081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.156056 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh2px\" (UniqueName: \"kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.257406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.257373 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.257616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.257430 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.257616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.257505 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh2px\" (UniqueName: \"kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.258178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.258151 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.259919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.259890 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.316191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.316095 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh2px\" (UniqueName: \"kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px\") pod \"maas-api-revoke-sub-e4e51282-zh64k\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.344056 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.344013 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:11.722679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:11.722643 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k"] Nov 02 17:43:11.723717 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:11.723690 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2ea409_d287_47d4_b327_17cbaefe300c.slice/crio-e722680191801c4baf7f22d419c69b0afce6b521d763756a87f34938fd465cf5 WatchSource:0}: Error finding container e722680191801c4baf7f22d419c69b0afce6b521d763756a87f34938fd465cf5: Status 404 returned error can't find the container with id e722680191801c4baf7f22d419c69b0afce6b521d763756a87f34938fd465cf5 Nov 02 17:43:12.481110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:12.481073 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" event={"ID":"0b2ea409-d287-47d4-b327-17cbaefe300c","Type":"ContainerStarted","Data":"8177f1c474a90292897cc1c78d6f538aea6989bbef8f5f8d0cfb90a995a46636"} Nov 02 17:43:12.481578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:12.481140 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" event={"ID":"0b2ea409-d287-47d4-b327-17cbaefe300c","Type":"ContainerStarted","Data":"e722680191801c4baf7f22d419c69b0afce6b521d763756a87f34938fd465cf5"} Nov 02 17:43:12.598890 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:12.598836 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" podStartSLOduration=2.598818884 podStartE2EDuration="2.598818884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:12.597758927 +0000 UTC m=+1179.671286128" watchObservedRunningTime="2026-11-02 17:43:12.598818884 +0000 UTC m=+1179.672346087" Nov 02 17:43:13.488670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.488636 5647 generic.go:356] "Generic (PLEG): container finished" podID="0b2ea409-d287-47d4-b327-17cbaefe300c" containerID="8177f1c474a90292897cc1c78d6f538aea6989bbef8f5f8d0cfb90a995a46636" exitCode=0 Nov 02 17:43:13.489122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.488724 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" event={"ID":"0b2ea409-d287-47d4-b327-17cbaefe300c","Type":"ContainerDied","Data":"8177f1c474a90292897cc1c78d6f538aea6989bbef8f5f8d0cfb90a995a46636"} Nov 02 17:43:13.797961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.797872 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb"] Nov 02 17:43:13.801816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.801795 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.858309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.858275 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb"] Nov 02 17:43:13.888782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.888723 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.888992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.888927 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.889074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.889051 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrwqk\" (UniqueName: \"kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.990030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.989994 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.990228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.990073 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrwqk\" (UniqueName: \"kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.990228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.990141 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.991092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.991069 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:13.992513 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:13.992493 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:14.048333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.048259 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrwqk\" (UniqueName: \"kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk\") pod \"maas-api-revoke-sub-e111751c-t55rb\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:14.114966 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.114931 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:14.490986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.490940 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb"] Nov 02 17:43:14.491434 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:14.491217 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9135108c_a7a8_49bf_b084_585428f886c5.slice/crio-245e1b02db2f2ec13574d90d3979982594855ca054b41cc238ec73cfa36598ce WatchSource:0}: Error finding container 245e1b02db2f2ec13574d90d3979982594855ca054b41cc238ec73cfa36598ce: Status 404 returned error can't find the container with id 245e1b02db2f2ec13574d90d3979982594855ca054b41cc238ec73cfa36598ce Nov 02 17:43:14.632199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.632169 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:14.698498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.698462 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px\" (UniqueName: \"kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px\") pod \"0b2ea409-d287-47d4-b327-17cbaefe300c\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " Nov 02 17:43:14.698658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.698569 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca\") pod \"0b2ea409-d287-47d4-b327-17cbaefe300c\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " Nov 02 17:43:14.698658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.698627 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca\") pod \"0b2ea409-d287-47d4-b327-17cbaefe300c\" (UID: \"0b2ea409-d287-47d4-b327-17cbaefe300c\") " Nov 02 17:43:14.699027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.699000 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca" pod "0b2ea409-d287-47d4-b327-17cbaefe300c" (UID: "0b2ea409-d287-47d4-b327-17cbaefe300c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:14.700902 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.700878 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca" pod "0b2ea409-d287-47d4-b327-17cbaefe300c" (UID: "0b2ea409-d287-47d4-b327-17cbaefe300c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:14.701016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.700898 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px" pod "0b2ea409-d287-47d4-b327-17cbaefe300c" (UID: "0b2ea409-d287-47d4-b327-17cbaefe300c"). InnerVolumeSpecName "kube-api-access-dh2px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:14.800149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.800064 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh2px\" (UniqueName: \"kubernetes.io/projected/0b2ea409-d287-47d4-b327-17cbaefe300c-kube-api-access-dh2px\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:14.800149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.800094 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:14.800149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.800104 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b2ea409-d287-47d4-b327-17cbaefe300c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:14.913614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.913570 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87"] Nov 02 17:43:14.914121 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.914106 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0b2ea409-d287-47d4-b327-17cbaefe300c" containerName="revoke-keys" Nov 02 17:43:14.914180 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.914122 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2ea409-d287-47d4-b327-17cbaefe300c" containerName="revoke-keys" Nov 02 17:43:14.914248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.914237 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2ea409-d287-47d4-b327-17cbaefe300c" containerName="revoke-keys" Nov 02 17:43:14.917635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.917615 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:14.973255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:14.973217 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87"] Nov 02 17:43:15.003444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.003404 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.003578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.003531 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5fl\" (UniqueName: \"kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.003652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.003626 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.104600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.104569 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.104797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.104623 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5fl\" (UniqueName: \"kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.104797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.104693 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.105394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.105369 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.107142 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.107116 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.162727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.162695 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5fl\" (UniqueName: \"kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl\") pod \"maas-api-revoke-sub-d8ce6a61-62m87\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.230512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.230481 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:15.503396 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.503368 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" Nov 02 17:43:15.503886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.503401 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k" event={"ID":"0b2ea409-d287-47d4-b327-17cbaefe300c","Type":"ContainerDied","Data":"e722680191801c4baf7f22d419c69b0afce6b521d763756a87f34938fd465cf5"} Nov 02 17:43:15.503886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.503463 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e722680191801c4baf7f22d419c69b0afce6b521d763756a87f34938fd465cf5" Nov 02 17:43:15.505339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.505310 5647 generic.go:356] "Generic (PLEG): container finished" podID="9135108c-a7a8-49bf-b084-585428f886c5" containerID="a9f9fd5381aea9a8ec8db8f228ec92ea8044bb810a41d62e4b1d24f43a0ecc5a" exitCode=0 Nov 02 17:43:15.505488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.505405 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" event={"ID":"9135108c-a7a8-49bf-b084-585428f886c5","Type":"ContainerDied","Data":"a9f9fd5381aea9a8ec8db8f228ec92ea8044bb810a41d62e4b1d24f43a0ecc5a"} Nov 02 17:43:15.505558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.505507 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" event={"ID":"9135108c-a7a8-49bf-b084-585428f886c5","Type":"ContainerStarted","Data":"245e1b02db2f2ec13574d90d3979982594855ca054b41cc238ec73cfa36598ce"} Nov 02 17:43:15.603869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:15.603841 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87"] Nov 02 17:43:15.607941 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:15.607867 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66ba264_a83a_4b03_b4a6_74fb21c29737.slice/crio-5821ae5a0310e348089d38783cd7a18c168a6d9cdcd10da2d6e8ca7659a8ad00 WatchSource:0}: Error finding container 5821ae5a0310e348089d38783cd7a18c168a6d9cdcd10da2d6e8ca7659a8ad00: Status 404 returned error can't find the container with id 5821ae5a0310e348089d38783cd7a18c168a6d9cdcd10da2d6e8ca7659a8ad00 Nov 02 17:43:16.513461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.513426 5647 generic.go:356] "Generic (PLEG): container finished" podID="b66ba264-a83a-4b03-b4a6-74fb21c29737" containerID="2f27d6aaa0152dccf40af3d03b2c363bf6ddf85cc127342cbc0beee69ebd7e72" exitCode=0 Nov 02 17:43:16.513899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.513525 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" event={"ID":"b66ba264-a83a-4b03-b4a6-74fb21c29737","Type":"ContainerDied","Data":"2f27d6aaa0152dccf40af3d03b2c363bf6ddf85cc127342cbc0beee69ebd7e72"} Nov 02 17:43:16.513899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.513609 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" event={"ID":"b66ba264-a83a-4b03-b4a6-74fb21c29737","Type":"ContainerStarted","Data":"5821ae5a0310e348089d38783cd7a18c168a6d9cdcd10da2d6e8ca7659a8ad00"} Nov 02 17:43:16.668478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.668453 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:16.825642 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.825561 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca\") pod \"9135108c-a7a8-49bf-b084-585428f886c5\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " Nov 02 17:43:16.825800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.825665 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca\") pod \"9135108c-a7a8-49bf-b084-585428f886c5\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " Nov 02 17:43:16.825855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.825804 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk\" (UniqueName: \"kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk\") pod \"9135108c-a7a8-49bf-b084-585428f886c5\" (UID: \"9135108c-a7a8-49bf-b084-585428f886c5\") " Nov 02 17:43:16.826042 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.826012 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca" pod "9135108c-a7a8-49bf-b084-585428f886c5" (UID: "9135108c-a7a8-49bf-b084-585428f886c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:16.826206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.826190 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:16.827820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.827798 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca" pod "9135108c-a7a8-49bf-b084-585428f886c5" (UID: "9135108c-a7a8-49bf-b084-585428f886c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:16.827976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.827960 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk" pod "9135108c-a7a8-49bf-b084-585428f886c5" (UID: "9135108c-a7a8-49bf-b084-585428f886c5"). InnerVolumeSpecName "kube-api-access-wrwqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:16.926937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.926902 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrwqk\" (UniqueName: \"kubernetes.io/projected/9135108c-a7a8-49bf-b084-585428f886c5-kube-api-access-wrwqk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:16.926937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:16.926930 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9135108c-a7a8-49bf-b084-585428f886c5-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:17.520996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.520916 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" Nov 02 17:43:17.520996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.520906 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb" event={"ID":"9135108c-a7a8-49bf-b084-585428f886c5","Type":"ContainerDied","Data":"245e1b02db2f2ec13574d90d3979982594855ca054b41cc238ec73cfa36598ce"} Nov 02 17:43:17.521448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.521014 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="245e1b02db2f2ec13574d90d3979982594855ca054b41cc238ec73cfa36598ce" Nov 02 17:43:17.669894 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.669863 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:17.837632 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.837599 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca\") pod \"b66ba264-a83a-4b03-b4a6-74fb21c29737\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " Nov 02 17:43:17.837817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.837754 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl\" (UniqueName: \"kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl\") pod \"b66ba264-a83a-4b03-b4a6-74fb21c29737\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " Nov 02 17:43:17.837899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.837813 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca\") pod \"b66ba264-a83a-4b03-b4a6-74fb21c29737\" (UID: \"b66ba264-a83a-4b03-b4a6-74fb21c29737\") " Nov 02 17:43:17.837977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.837953 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca" pod "b66ba264-a83a-4b03-b4a6-74fb21c29737" (UID: "b66ba264-a83a-4b03-b4a6-74fb21c29737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:17.838247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.838229 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:17.840023 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.839988 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca" pod "b66ba264-a83a-4b03-b4a6-74fb21c29737" (UID: "b66ba264-a83a-4b03-b4a6-74fb21c29737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:17.840129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.840109 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl" pod "b66ba264-a83a-4b03-b4a6-74fb21c29737" (UID: "b66ba264-a83a-4b03-b4a6-74fb21c29737"). InnerVolumeSpecName "kube-api-access-6x5fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:17.939963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.939924 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b66ba264-a83a-4b03-b4a6-74fb21c29737-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:17.939963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:17.939956 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5fl\" (UniqueName: \"kubernetes.io/projected/b66ba264-a83a-4b03-b4a6-74fb21c29737-kube-api-access-6x5fl\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:18.047976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:18.047938 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k"] Nov 02 17:43:18.050620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:18.050590 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ded7d84-sqh2k"] Nov 02 17:43:18.529844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:18.529814 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" event={"ID":"b66ba264-a83a-4b03-b4a6-74fb21c29737","Type":"ContainerDied","Data":"5821ae5a0310e348089d38783cd7a18c168a6d9cdcd10da2d6e8ca7659a8ad00"} Nov 02 17:43:18.530199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:18.529849 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87" Nov 02 17:43:18.530199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:18.529859 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5821ae5a0310e348089d38783cd7a18c168a6d9cdcd10da2d6e8ca7659a8ad00" Nov 02 17:43:18.602178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:18.602132 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:43:19.081217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:19.081181 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1aef59-f21d-4f43-8ac9-de071d307858" path="/var/lib/kubelet/pods/5a1aef59-f21d-4f43-8ac9-de071d307858/volumes" Nov 02 17:43:20.058761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:20.057208 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws"] Nov 02 17:43:20.183756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:20.182899 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcc889c6-g2sws"] Nov 02 17:43:21.119809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:21.118657 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6bf8309-a7fa-4b92-a59f-cbad489d0bc2" path="/var/lib/kubelet/pods/a6bf8309-a7fa-4b92-a59f-cbad489d0bc2/volumes" Nov 02 17:43:24.526841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.526810 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8"] Nov 02 17:43:24.527352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.527338 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b66ba264-a83a-4b03-b4a6-74fb21c29737" containerName="revoke-keys" Nov 02 17:43:24.527392 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.527359 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66ba264-a83a-4b03-b4a6-74fb21c29737" containerName="revoke-keys" Nov 02 17:43:24.527392 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.527388 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9135108c-a7a8-49bf-b084-585428f886c5" containerName="revoke-keys" Nov 02 17:43:24.527469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.527395 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9135108c-a7a8-49bf-b084-585428f886c5" containerName="revoke-keys" Nov 02 17:43:24.527498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.527477 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9135108c-a7a8-49bf-b084-585428f886c5" containerName="revoke-keys" Nov 02 17:43:24.527498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.527490 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66ba264-a83a-4b03-b4a6-74fb21c29737" containerName="revoke-keys" Nov 02 17:43:24.531921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.531902 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.589927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.589896 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8"] Nov 02 17:43:24.636459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.636418 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:24.636612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.636499 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:43:24.704799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.704723 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.704984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.704824 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.705038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.705002 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfq8h\" (UniqueName: \"kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.806691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.806590 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfq8h\" (UniqueName: \"kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.806691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.806663 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.806960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.806717 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.807438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.807408 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.809269 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.809232 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:24.865489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:24.865452 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfq8h\" (UniqueName: \"kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h\") pod \"maas-api-revoke-sub-df2e2ced-jc4p8\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:25.144504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:25.144465 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:25.521330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:25.521302 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8"] Nov 02 17:43:25.522369 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:25.522330 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6804dc1_e6a7_445e_a369_43cd784d64e6.slice/crio-d572df0f7e7f0243d0466560368284ae44842b6b8bc841b8aa9273527de52331 WatchSource:0}: Error finding container d572df0f7e7f0243d0466560368284ae44842b6b8bc841b8aa9273527de52331: Status 404 returned error can't find the container with id d572df0f7e7f0243d0466560368284ae44842b6b8bc841b8aa9273527de52331 Nov 02 17:43:25.617330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:25.617291 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" event={"ID":"a6804dc1-e6a7-445e-a369-43cd784d64e6","Type":"ContainerStarted","Data":"d572df0f7e7f0243d0466560368284ae44842b6b8bc841b8aa9273527de52331"} Nov 02 17:43:26.625325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:26.625293 5647 generic.go:356] "Generic (PLEG): container finished" podID="a6804dc1-e6a7-445e-a369-43cd784d64e6" containerID="cd81187ebd33d7dd1882edada5729e7f92fd23b945824fac56ea58579edee9d1" exitCode=0 Nov 02 17:43:26.625779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:26.625331 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" event={"ID":"a6804dc1-e6a7-445e-a369-43cd784d64e6","Type":"ContainerDied","Data":"cd81187ebd33d7dd1882edada5729e7f92fd23b945824fac56ea58579edee9d1"} Nov 02 17:43:27.792137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.792108 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:27.946337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.946249 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h\" (UniqueName: \"kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h\") pod \"a6804dc1-e6a7-445e-a369-43cd784d64e6\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " Nov 02 17:43:27.946337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.946311 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca\") pod \"a6804dc1-e6a7-445e-a369-43cd784d64e6\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " Nov 02 17:43:27.946577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.946443 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca\") pod \"a6804dc1-e6a7-445e-a369-43cd784d64e6\" (UID: \"a6804dc1-e6a7-445e-a369-43cd784d64e6\") " Nov 02 17:43:27.946682 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.946654 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca" pod "a6804dc1-e6a7-445e-a369-43cd784d64e6" (UID: "a6804dc1-e6a7-445e-a369-43cd784d64e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:27.946996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.946972 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:27.948656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.948630 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca" pod "a6804dc1-e6a7-445e-a369-43cd784d64e6" (UID: "a6804dc1-e6a7-445e-a369-43cd784d64e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:27.948783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:27.948698 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h" pod "a6804dc1-e6a7-445e-a369-43cd784d64e6" (UID: "a6804dc1-e6a7-445e-a369-43cd784d64e6"). InnerVolumeSpecName "kube-api-access-jfq8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:28.047899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:28.047859 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6804dc1-e6a7-445e-a369-43cd784d64e6-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:28.047899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:28.047893 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfq8h\" (UniqueName: \"kubernetes.io/projected/a6804dc1-e6a7-445e-a369-43cd784d64e6-kube-api-access-jfq8h\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:28.638672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:28.638644 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" Nov 02 17:43:28.638914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:28.638639 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8" event={"ID":"a6804dc1-e6a7-445e-a369-43cd784d64e6","Type":"ContainerDied","Data":"d572df0f7e7f0243d0466560368284ae44842b6b8bc841b8aa9273527de52331"} Nov 02 17:43:28.638914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:28.638813 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d572df0f7e7f0243d0466560368284ae44842b6b8bc841b8aa9273527de52331" Nov 02 17:43:29.381249 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.381213 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7"] Nov 02 17:43:29.381771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.381756 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a6804dc1-e6a7-445e-a369-43cd784d64e6" containerName="revoke-keys" Nov 02 17:43:29.381823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.381774 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6804dc1-e6a7-445e-a369-43cd784d64e6" containerName="revoke-keys" Nov 02 17:43:29.381898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.381888 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6804dc1-e6a7-445e-a369-43cd784d64e6" containerName="revoke-keys" Nov 02 17:43:29.385299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.385284 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.438715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.438682 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:29.489823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.489793 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:43:29.494354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.494322 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7"] Nov 02 17:43:29.565940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.565901 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nqn\" (UniqueName: \"kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.566157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.566041 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.566157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.566106 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.667654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.667568 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.667843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.667653 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.667843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.667779 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nqn\" (UniqueName: \"kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.668498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.668468 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.670237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.670212 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.725873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.725841 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nqn\" (UniqueName: \"kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn\") pod \"maas-api-revoke-sub-4c305d2c-gsqt7\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:29.998816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:29.998712 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:30.382232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.382194 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7"] Nov 02 17:43:30.383145 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:30.383119 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0 WatchSource:0}: Error finding container ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0: Status 404 returned error can't find the container with id ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0 Nov 02 17:43:30.652748 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.652646 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" event={"ID":"1116c286-a714-43f3-bc7a-99e8e19d7cfe","Type":"ContainerStarted","Data":"ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0"} Nov 02 17:43:30.797871 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.797840 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl"] Nov 02 17:43:30.801960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.801928 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.868575 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.868526 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq"] Nov 02 17:43:30.882040 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.881041 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.882761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.882689 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.882921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.882805 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.882921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.882884 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.883038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.882938 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.888045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.888016 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.891458 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.891421 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl"] Nov 02 17:43:30.953532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.953496 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq"] Nov 02 17:43:30.983901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.983866 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.983901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.983905 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.983928 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.983972 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.983997 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984025 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984156 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984207 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kcz5\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984240 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984283 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984297 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984317 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984335 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984411 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984519 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984575 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzdk2\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984617 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.984804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984661 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984772 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:30.984929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.984816 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.985068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.985053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:30.986703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:30.986679 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.085404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085367 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.085576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085437 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.085576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085469 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.085694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085594 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.085694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085653 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.085820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085761 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kcz5\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.085820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085795 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.085925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085848 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.085925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085887 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.085948 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086037 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzdk2\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086049 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086072 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086246 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086090 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086246 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086128 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086243 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.086354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086280 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086464 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.086925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.086904 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.088375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.088346 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.088632 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.088614 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.088698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.088653 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.143897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.143866 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.144056 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.143997 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.144102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.144076 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzdk2\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.144145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.144124 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kcz5\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5\") pod \"e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.221144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.221063 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:31.415859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.415824 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:31.628357 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.628317 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc"] Nov 02 17:43:31.633445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.633418 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.654570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.654533 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq"] Nov 02 17:43:31.656041 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:31.656016 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57dcf88a_e0b4_486e_8ec8_cbc04e075767.slice/crio-54b904eed89896ec979e995414f0c49dd02152bf63b561853b0e9cd1ccc41fb4 WatchSource:0}: Error finding container 54b904eed89896ec979e995414f0c49dd02152bf63b561853b0e9cd1ccc41fb4: Status 404 returned error can't find the container with id 54b904eed89896ec979e995414f0c49dd02152bf63b561853b0e9cd1ccc41fb4 Nov 02 17:43:31.659509 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.659468 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:31.659619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.659553 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:31.659619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.659602 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:31.663412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.663303 5647 generic.go:356] "Generic (PLEG): container finished" podID="1116c286-a714-43f3-bc7a-99e8e19d7cfe" containerID="4bba6696efed6c9d2b3310b67cee167cd370c72b942b4b54ce56e9c874dc6e08" exitCode=0 Nov 02 17:43:31.663412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.663354 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" event={"ID":"1116c286-a714-43f3-bc7a-99e8e19d7cfe","Type":"ContainerDied","Data":"4bba6696efed6c9d2b3310b67cee167cd370c72b942b4b54ce56e9c874dc6e08"} Nov 02 17:43:31.688108 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.688071 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a099c3\"/\"distinct-2-w2-a099c3-kserve-self-signed-certs\"" Nov 02 17:43:31.691843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.691811 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8qr9\" (UniqueName: \"kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.692020 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.691984 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.692089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.692021 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.692089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.692049 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.692194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.692120 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc"] Nov 02 17:43:31.692287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.692266 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.692364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.692324 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.797753 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl"] Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.798838 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.798901 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.798958 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8qr9\" (UniqueName: \"kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.799077 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.799101 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.801768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.799126 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.804889 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.802757 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.804889 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.803008 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.804889 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.803234 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.816371 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:31.814370 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9fd3401_6d47_40fb_8333_db635f95f0ed.slice/crio-4769a4b20ed7095afe2207ca1959cc555eda600b1ed5e5fce7f41eba42465af8 WatchSource:0}: Error finding container 4769a4b20ed7095afe2207ca1959cc555eda600b1ed5e5fce7f41eba42465af8: Status 404 returned error can't find the container with id 4769a4b20ed7095afe2207ca1959cc555eda600b1ed5e5fce7f41eba42465af8 Nov 02 17:43:31.820461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.819851 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:31.820461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.819955 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:31.820461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.820011 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:31.849924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.849886 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.859618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.859547 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.864771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.863258 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8qr9\" (UniqueName: \"kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9\") pod \"distinct-2-w2-a099c3-kserve-859f8899df-tmglc\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:31.946259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:31.946231 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:32.402912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.402882 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc"] Nov 02 17:43:32.403588 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:32.403561 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f5bbe3c_b8e2_47b4_bab9_651eecceed79.slice/crio-fe5d6a8c42354fb8f9e1af4c8d77f58c10cacb4e45d4dcff66295afd61880be5 WatchSource:0}: Error finding container fe5d6a8c42354fb8f9e1af4c8d77f58c10cacb4e45d4dcff66295afd61880be5: Status 404 returned error can't find the container with id fe5d6a8c42354fb8f9e1af4c8d77f58c10cacb4e45d4dcff66295afd61880be5 Nov 02 17:43:32.679859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.679726 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" event={"ID":"57dcf88a-e0b4-486e-8ec8-cbc04e075767","Type":"ContainerStarted","Data":"150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5"} Nov 02 17:43:32.680222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.679875 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" event={"ID":"57dcf88a-e0b4-486e-8ec8-cbc04e075767","Type":"ContainerStarted","Data":"54b904eed89896ec979e995414f0c49dd02152bf63b561853b0e9cd1ccc41fb4"} Nov 02 17:43:32.683141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.683116 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" event={"ID":"e9fd3401-6d47-40fb-8333-db635f95f0ed","Type":"ContainerStarted","Data":"76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8"} Nov 02 17:43:32.683315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.683302 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" event={"ID":"e9fd3401-6d47-40fb-8333-db635f95f0ed","Type":"ContainerStarted","Data":"4769a4b20ed7095afe2207ca1959cc555eda600b1ed5e5fce7f41eba42465af8"} Nov 02 17:43:32.685719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.685693 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" event={"ID":"0f5bbe3c-b8e2-47b4-bab9-651eecceed79","Type":"ContainerStarted","Data":"35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7"} Nov 02 17:43:32.685935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.685914 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" event={"ID":"0f5bbe3c-b8e2-47b4-bab9-651eecceed79","Type":"ContainerStarted","Data":"fe5d6a8c42354fb8f9e1af4c8d77f58c10cacb4e45d4dcff66295afd61880be5"} Nov 02 17:43:32.957504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.955363 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:32.960674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.958761 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podStartSLOduration=2.958720296 podStartE2EDuration="2.958720296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.829062917 +0000 UTC m=+1199.902590121" watchObservedRunningTime="2026-11-02 17:43:32.958720296 +0000 UTC m=+1200.032247499" Nov 02 17:43:32.960674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.959873 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podStartSLOduration=1.959862021 podStartE2EDuration="1.959862021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.958182968 +0000 UTC m=+1200.031710181" watchObservedRunningTime="2026-11-02 17:43:32.959862021 +0000 UTC m=+1200.033389227" Nov 02 17:43:32.966787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:32.966764 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:33.062425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.061782 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn\" (UniqueName: \"kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn\") pod \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " Nov 02 17:43:33.062425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.061852 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca\") pod \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " Nov 02 17:43:33.062425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.062012 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca\") pod \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\" (UID: \"1116c286-a714-43f3-bc7a-99e8e19d7cfe\") " Nov 02 17:43:33.066107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.064278 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca" pod "1116c286-a714-43f3-bc7a-99e8e19d7cfe" (UID: "1116c286-a714-43f3-bc7a-99e8e19d7cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:33.066107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.065469 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:33.196761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.188936 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca" pod "1116c286-a714-43f3-bc7a-99e8e19d7cfe" (UID: "1116c286-a714-43f3-bc7a-99e8e19d7cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:33.196761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.194380 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1116c286-a714-43f3-bc7a-99e8e19d7cfe-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:33.200440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.200400 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn" pod "1116c286-a714-43f3-bc7a-99e8e19d7cfe" (UID: "1116c286-a714-43f3-bc7a-99e8e19d7cfe"). InnerVolumeSpecName "kube-api-access-w8nqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:33.263161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.257266 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:33.263161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.259919 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 17:43:33.263161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.259979 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 17:43:33.295783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.295281 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nqn\" (UniqueName: \"kubernetes.io/projected/1116c286-a714-43f3-bc7a-99e8e19d7cfe-kube-api-access-w8nqn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:33.314554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.313828 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podStartSLOduration=3.313794692 podStartE2EDuration="3.313794692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:33.144855888 +0000 UTC m=+1200.218383091" watchObservedRunningTime="2026-11-02 17:43:33.313794692 +0000 UTC m=+1200.387321897" Nov 02 17:43:33.373340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.373220 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:43:33.380766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.379202 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:43:33.430862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.430813 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:33.454611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.452973 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:43:33.454611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.453055 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:43:33.695177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.695125 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" Nov 02 17:43:33.696727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.696688 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7" event={"ID":"1116c286-a714-43f3-bc7a-99e8e19d7cfe","Type":"ContainerDied","Data":"ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0"} Nov 02 17:43:33.696922 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:33.696795 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0" Nov 02 17:43:34.054829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:34.053816 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx"] Nov 02 17:43:34.058766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:34.058692 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85652222-8g7sx"] Nov 02 17:43:34.232763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:34.224224 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 17:43:34.232763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:34.224289 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 17:43:34.431920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:34.431879 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:43:34.432215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:34.432183 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:43:35.081256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:35.081218 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a17321-eccc-45cc-923e-2f981fb59aa6" path="/var/lib/kubelet/pods/c5a17321-eccc-45cc-923e-2f981fb59aa6/volumes" Nov 02 17:43:35.225762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:35.222921 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 17:43:35.225762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:35.222996 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 17:43:35.422761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:35.419162 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:43:35.422761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:35.419223 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:43:35.994953 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:35.994915 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:43:36.226762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.224186 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 17:43:36.226762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.224266 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 17:43:36.428760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.426727 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:43:36.428760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.426842 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:43:36.861890 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.861652 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb"] Nov 02 17:43:36.863448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.863414 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1116c286-a714-43f3-bc7a-99e8e19d7cfe" containerName="revoke-keys" Nov 02 17:43:36.863648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.863632 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1116c286-a714-43f3-bc7a-99e8e19d7cfe" containerName="revoke-keys" Nov 02 17:43:36.864130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.863978 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1116c286-a714-43f3-bc7a-99e8e19d7cfe" containerName="revoke-keys" Nov 02 17:43:36.869778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.869759 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:36.895842 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:36.895808 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb"] Nov 02 17:43:37.001518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001483 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.001708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001557 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.001708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001587 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.010015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001707 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.010015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001833 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.010015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001864 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.010015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001910 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvdpn\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.010015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001941 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.010015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.001984 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111484 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111536 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111581 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvdpn\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111615 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111657 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111717 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111791 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111817 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.114808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.111955 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.131030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.130478 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.131030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.130747 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.131030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.130970 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.135052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.135026 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.135263 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.135245 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.155434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.155387 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.160644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.160615 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.169311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.169239 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.178934 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.178905 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvdpn\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn\") pod \"e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.184398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.184376 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:37.250758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.242868 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 17:43:37.250758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.242938 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 17:43:37.458026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.457903 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:43:37.458217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.458018 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:43:37.672778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.672726 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb"] Nov 02 17:43:37.674871 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.674834 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:37.674949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.674911 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:37.674983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.674947 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:43:37.775176 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:37.775001 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:43:37.780771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:37.778132 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" event={"ID":"af0eba83-6e44-47a3-8315-4b8eb360ed15","Type":"ContainerStarted","Data":"f2ef81d9a6805d85679a8b47676d31ea946cfd97a4882489986fc5a89e34da41"} Nov 02 17:43:38.222492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:38.222454 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 02 17:43:38.222725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:38.222519 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 02 17:43:38.416726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:38.416680 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Nov 02 17:43:38.417135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:38.416781 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Nov 02 17:43:38.790405 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:38.790359 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" event={"ID":"af0eba83-6e44-47a3-8315-4b8eb360ed15","Type":"ContainerStarted","Data":"7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807"} Nov 02 17:43:38.917765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:38.916662 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podStartSLOduration=2.916643348 podStartE2EDuration="2.916643348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:38.916473419 +0000 UTC m=+1205.990000625" watchObservedRunningTime="2026-11-02 17:43:38.916643348 +0000 UTC m=+1205.990170552" Nov 02 17:43:39.185229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.185184 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:39.187232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.187201 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:39.187389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.187263 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:39.662624 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.662480 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:39.663292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.663253 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:39.664059 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.664039 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:39.664191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.664176 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:39.664898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.664876 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:43:39.665049 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:39.665036 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:43:40.056244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.056159 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv"] Nov 02 17:43:40.068278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.068244 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9338cdef-8g7bv"] Nov 02 17:43:40.133433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.133395 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk"] Nov 02 17:43:40.138486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.138461 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.190193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.190157 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:40.190463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.190428 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:40.191930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.191905 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:40.192270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.192249 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:43:40.194954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.194929 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk"] Nov 02 17:43:40.211784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.202216 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.211784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.202423 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.211784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.202511 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w9ks\" (UniqueName: \"kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.309820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.309295 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w9ks\" (UniqueName: \"kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.309820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.309384 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.309820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.309592 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.326762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.310467 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.328493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.328455 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.367901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.367871 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w9ks\" (UniqueName: \"kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks\") pod \"maas-api-revoke-sub-7ecf78dd-zbdgk\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.455050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.455017 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:40.883139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:40.880928 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk"] Nov 02 17:43:41.081534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.081503 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abef49e-37f5-442a-9881-769d93359722" path="/var/lib/kubelet/pods/9abef49e-37f5-442a-9881-769d93359722/volumes" Nov 02 17:43:41.185168 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.185133 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:41.185352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.185193 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:41.882885 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.882844 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" event={"ID":"b07b8156-7c4d-45e8-9fe3-137a3b386f8d","Type":"ContainerStarted","Data":"0f69a99ec5475e0a6416170ee8629f3cec01ed2221cb870c83cf8fc3c26a3206"} Nov 02 17:43:41.883108 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.882915 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" event={"ID":"b07b8156-7c4d-45e8-9fe3-137a3b386f8d","Type":"ContainerStarted","Data":"e15c8de7c08bafef6e0c75df53d1442d362ac77fe2188f71db5ccc1d8c5f5607"} Nov 02 17:43:41.947390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.947356 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:41.959821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:41.959792 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:42.019412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.002648 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" podStartSLOduration=2.002626455 podStartE2EDuration="2.002626455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:41.997430473 +0000 UTC m=+1209.070957676" watchObservedRunningTime="2026-11-02 17:43:42.002626455 +0000 UTC m=+1209.076153720" Nov 02 17:43:42.185507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.185423 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:42.185507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.185489 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:42.430835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.430789 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj"] Nov 02 17:43:42.434678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.434657 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.465199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.465154 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.465367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.465221 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.465367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.465262 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgtz\" (UniqueName: \"kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.491275 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.491242 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj"] Nov 02 17:43:42.566709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.566675 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.566926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.566721 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.566926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.566789 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgtz\" (UniqueName: \"kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.567518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.567491 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.569341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.569319 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.631634 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.631590 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgtz\" (UniqueName: \"kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz\") pod \"maas-api-revoke-sub-423c9af2-hjgfj\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.749903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.749822 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:42.894271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.894223 5647 generic.go:356] "Generic (PLEG): container finished" podID="b07b8156-7c4d-45e8-9fe3-137a3b386f8d" containerID="0f69a99ec5475e0a6416170ee8629f3cec01ed2221cb870c83cf8fc3c26a3206" exitCode=0 Nov 02 17:43:42.895000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.894579 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" event={"ID":"b07b8156-7c4d-45e8-9fe3-137a3b386f8d","Type":"ContainerDied","Data":"0f69a99ec5475e0a6416170ee8629f3cec01ed2221cb870c83cf8fc3c26a3206"} Nov 02 17:43:42.944953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:42.944916 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:43:43.166997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:43.166965 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj"] Nov 02 17:43:43.185454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:43.185426 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:43.185608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:43.185476 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:43.965681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:43.965644 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" event={"ID":"a5b58511-d066-45b6-825f-8a6ebbd4a989","Type":"ContainerStarted","Data":"9363eb0db204d831960fa8f58288f1a43bd5f2b6afd2a511ce6bc18ee6383bf6"} Nov 02 17:43:43.976011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:43.975967 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" event={"ID":"a5b58511-d066-45b6-825f-8a6ebbd4a989","Type":"ContainerStarted","Data":"a7b2eb2244316f9dd8362057e4ba3a3e6559fd267e264fa91bec07128e019701"} Nov 02 17:43:44.123332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:44.123267 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" podStartSLOduration=2.123243219 podStartE2EDuration="2.123243219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:44.113035303 +0000 UTC m=+1211.186562506" watchObservedRunningTime="2026-11-02 17:43:44.123243219 +0000 UTC m=+1211.196770618" Nov 02 17:43:44.186435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:44.186341 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:44.187044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:44.186996 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:45.216914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.205057 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:45.216914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.212856 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:45.348776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.329785 5647 generic.go:356] "Generic (PLEG): container finished" podID="a5b58511-d066-45b6-825f-8a6ebbd4a989" containerID="9363eb0db204d831960fa8f58288f1a43bd5f2b6afd2a511ce6bc18ee6383bf6" exitCode=0 Nov 02 17:43:45.367623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.367581 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" event={"ID":"a5b58511-d066-45b6-825f-8a6ebbd4a989","Type":"ContainerDied","Data":"9363eb0db204d831960fa8f58288f1a43bd5f2b6afd2a511ce6bc18ee6383bf6"} Nov 02 17:43:45.394305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.394261 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:45.467760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.467636 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 17:43:45.482892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.482843 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:45.496196 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.495119 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks\" (UniqueName: \"kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks\") pod \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " Nov 02 17:43:45.496196 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.495366 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca\") pod \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " Nov 02 17:43:45.496196 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.495436 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca\") pod \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\" (UID: \"b07b8156-7c4d-45e8-9fe3-137a3b386f8d\") " Nov 02 17:43:45.496992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.496653 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca" pod "b07b8156-7c4d-45e8-9fe3-137a3b386f8d" (UID: "b07b8156-7c4d-45e8-9fe3-137a3b386f8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:45.512762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.510428 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks" pod "b07b8156-7c4d-45e8-9fe3-137a3b386f8d" (UID: "b07b8156-7c4d-45e8-9fe3-137a3b386f8d"). InnerVolumeSpecName "kube-api-access-5w9ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:45.512762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.510893 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca" pod "b07b8156-7c4d-45e8-9fe3-137a3b386f8d" (UID: "b07b8156-7c4d-45e8-9fe3-137a3b386f8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:45.598706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.598670 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:45.598970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.598954 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:45.599091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.599078 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w9ks\" (UniqueName: \"kubernetes.io/projected/b07b8156-7c4d-45e8-9fe3-137a3b386f8d-kube-api-access-5w9ks\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:45.722813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.722059 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:45.957139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:45.930524 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 17:43:46.270001 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.269957 5647 patch_prober.go:29] interesting pod/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 02 17:43:46.310002 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.309950 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 02 17:43:46.738034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.737989 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:43:46.739150 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.739125 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:43:46.806979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.769751 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" Nov 02 17:43:46.817773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.806993 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk" event={"ID":"b07b8156-7c4d-45e8-9fe3-137a3b386f8d","Type":"ContainerDied","Data":"e15c8de7c08bafef6e0c75df53d1442d362ac77fe2188f71db5ccc1d8c5f5607"} Nov 02 17:43:46.817773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:46.814994 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e15c8de7c08bafef6e0c75df53d1442d362ac77fe2188f71db5ccc1d8c5f5607" Nov 02 17:43:47.258640 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.257966 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:43:47.366601 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:47.366556 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:43:47.416031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.415986 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:47.516812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.516721 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:47.517288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.517261 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:47.518388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.518365 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:43:47.552904 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.551980 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz\" (UniqueName: \"kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz\") pod \"a5b58511-d066-45b6-825f-8a6ebbd4a989\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " Nov 02 17:43:47.553207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.553184 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca\") pod \"a5b58511-d066-45b6-825f-8a6ebbd4a989\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " Nov 02 17:43:47.554286 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.554222 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca\") pod \"a5b58511-d066-45b6-825f-8a6ebbd4a989\" (UID: \"a5b58511-d066-45b6-825f-8a6ebbd4a989\") " Nov 02 17:43:47.554492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.553830 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca" pod "a5b58511-d066-45b6-825f-8a6ebbd4a989" (UID: "a5b58511-d066-45b6-825f-8a6ebbd4a989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:47.555385 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.555363 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:47.556300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.556272 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz" pod "a5b58511-d066-45b6-825f-8a6ebbd4a989" (UID: "a5b58511-d066-45b6-825f-8a6ebbd4a989"). InnerVolumeSpecName "kube-api-access-dfgtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:47.563123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.563096 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca" pod "a5b58511-d066-45b6-825f-8a6ebbd4a989" (UID: "a5b58511-d066-45b6-825f-8a6ebbd4a989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:47.656712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.656662 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfgtz\" (UniqueName: \"kubernetes.io/projected/a5b58511-d066-45b6-825f-8a6ebbd4a989-kube-api-access-dfgtz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:47.656712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.656715 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5b58511-d066-45b6-825f-8a6ebbd4a989-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:47.790168 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.789068 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" Nov 02 17:43:47.790168 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.789187 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj" event={"ID":"a5b58511-d066-45b6-825f-8a6ebbd4a989","Type":"ContainerDied","Data":"a7b2eb2244316f9dd8362057e4ba3a3e6559fd267e264fa91bec07128e019701"} Nov 02 17:43:47.790168 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:47.789241 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7b2eb2244316f9dd8362057e4ba3a3e6559fd267e264fa91bec07128e019701" Nov 02 17:43:47.792294 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:47.792256 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:43:50.046311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:50.046269 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x"] Nov 02 17:43:50.096674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:50.096641 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9aee483-2cf8x"] Nov 02 17:43:51.081418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:51.081386 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c37c61-8bd9-4cbe-906a-bd1df943a1fb" path="/var/lib/kubelet/pods/a0c37c61-8bd9-4cbe-906a-bd1df943a1fb/volumes" Nov 02 17:43:53.162199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162157 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67"] Nov 02 17:43:53.162696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162681 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a5b58511-d066-45b6-825f-8a6ebbd4a989" containerName="revoke-keys" Nov 02 17:43:53.162785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162698 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b58511-d066-45b6-825f-8a6ebbd4a989" containerName="revoke-keys" Nov 02 17:43:53.162785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162751 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b07b8156-7c4d-45e8-9fe3-137a3b386f8d" containerName="revoke-keys" Nov 02 17:43:53.162785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162756 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07b8156-7c4d-45e8-9fe3-137a3b386f8d" containerName="revoke-keys" Nov 02 17:43:53.162897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162834 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b58511-d066-45b6-825f-8a6ebbd4a989" containerName="revoke-keys" Nov 02 17:43:53.162897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.162848 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07b8156-7c4d-45e8-9fe3-137a3b386f8d" containerName="revoke-keys" Nov 02 17:43:53.166437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.166420 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.222321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.222289 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67"] Nov 02 17:43:53.229371 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.229344 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.229525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.229482 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.229668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.229649 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv8g\" (UniqueName: \"kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.269468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.269439 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:53.269619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.269483 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:43:53.331055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.331022 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.331270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.331120 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.331270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.331254 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv8g\" (UniqueName: \"kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.331910 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.331848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.333591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.333572 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.389169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.389137 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv8g\" (UniqueName: \"kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g\") pod \"maas-api-revoke-sub-9b55dbfe-sbx67\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.479954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.479879 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:53.863854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:53.863823 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67"] Nov 02 17:43:53.864869 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:53.864845 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb9f49f_3988_4692_86fe_e4ee96c5ec05.slice/crio-f862ddbd03798cb205147c12202d926a5b853f2ea27ef83602393b438c5dbd32 WatchSource:0}: Error finding container f862ddbd03798cb205147c12202d926a5b853f2ea27ef83602393b438c5dbd32: Status 404 returned error can't find the container with id f862ddbd03798cb205147c12202d926a5b853f2ea27ef83602393b438c5dbd32 Nov 02 17:43:54.836427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:54.836394 5647 generic.go:356] "Generic (PLEG): container finished" podID="ecb9f49f-3988-4692-86fe-e4ee96c5ec05" containerID="9240734f47463a39627edf65e2adf16fa760bbf8cc0b47214d82bcda61d64c26" exitCode=0 Nov 02 17:43:54.836815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:54.836439 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" event={"ID":"ecb9f49f-3988-4692-86fe-e4ee96c5ec05","Type":"ContainerDied","Data":"9240734f47463a39627edf65e2adf16fa760bbf8cc0b47214d82bcda61d64c26"} Nov 02 17:43:54.836815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:54.836513 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" event={"ID":"ecb9f49f-3988-4692-86fe-e4ee96c5ec05","Type":"ContainerStarted","Data":"f862ddbd03798cb205147c12202d926a5b853f2ea27ef83602393b438c5dbd32"} Nov 02 17:43:55.982327 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:55.982293 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:43:55.982648 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:55.982396 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache]" Nov 02 17:43:55.998017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:55.997990 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:56.068193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.068162 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca\") pod \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " Nov 02 17:43:56.068373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.068348 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca\") pod \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " Nov 02 17:43:56.068442 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.068427 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g\" (UniqueName: \"kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g\") pod \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\" (UID: \"ecb9f49f-3988-4692-86fe-e4ee96c5ec05\") " Nov 02 17:43:56.068515 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.068491 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca" pod "ecb9f49f-3988-4692-86fe-e4ee96c5ec05" (UID: "ecb9f49f-3988-4692-86fe-e4ee96c5ec05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:43:56.068911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.068884 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:56.070679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.070651 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g" pod "ecb9f49f-3988-4692-86fe-e4ee96c5ec05" (UID: "ecb9f49f-3988-4692-86fe-e4ee96c5ec05"). InnerVolumeSpecName "kube-api-access-slv8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:43:56.070787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.070691 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca" pod "ecb9f49f-3988-4692-86fe-e4ee96c5ec05" (UID: "ecb9f49f-3988-4692-86fe-e4ee96c5ec05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:43:56.169958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.169870 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:56.169958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.169904 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv8g\" (UniqueName: \"kubernetes.io/projected/ecb9f49f-3988-4692-86fe-e4ee96c5ec05-kube-api-access-slv8g\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:43:56.851619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.851586 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" Nov 02 17:43:56.851830 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.851619 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67" event={"ID":"ecb9f49f-3988-4692-86fe-e4ee96c5ec05","Type":"ContainerDied","Data":"f862ddbd03798cb205147c12202d926a5b853f2ea27ef83602393b438c5dbd32"} Nov 02 17:43:56.851830 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:56.851664 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f862ddbd03798cb205147c12202d926a5b853f2ea27ef83602393b438c5dbd32" Nov 02 17:43:57.814410 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:43:57.814359 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache]" Nov 02 17:43:58.777285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.777250 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n"] Nov 02 17:43:58.777818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.777803 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ecb9f49f-3988-4692-86fe-e4ee96c5ec05" containerName="revoke-keys" Nov 02 17:43:58.777877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.777819 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb9f49f-3988-4692-86fe-e4ee96c5ec05" containerName="revoke-keys" Nov 02 17:43:58.777959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.777949 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb9f49f-3988-4692-86fe-e4ee96c5ec05" containerName="revoke-keys" Nov 02 17:43:58.781917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.781902 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:58.837900 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.837865 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n"] Nov 02 17:43:58.884854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.884818 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:43:58.884854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.884833 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:43:58.902725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.902687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:58.902968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.902830 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:58.903275 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:58.903250 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dktf\" (UniqueName: \"kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.004308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.004271 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dktf\" (UniqueName: \"kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.004489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.004345 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.004489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.004409 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.005165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.005082 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.006982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.006952 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.062677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.062597 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dktf\" (UniqueName: \"kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf\") pod \"maas-api-revoke-sub-a8487438-wv88n\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.094434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.094403 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:43:59.468764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.468711 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n"] Nov 02 17:43:59.469518 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:43:59.469490 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7743d48e_e421_44ee_952c_97424e208e4c.slice/crio-d0ec8966fcbdcb249e8a1ed16aa804dee0fef9ffdd06f4115772b572051b35f3 WatchSource:0}: Error finding container d0ec8966fcbdcb249e8a1ed16aa804dee0fef9ffdd06f4115772b572051b35f3: Status 404 returned error can't find the container with id d0ec8966fcbdcb249e8a1ed16aa804dee0fef9ffdd06f4115772b572051b35f3 Nov 02 17:43:59.790882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.790848 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z"] Nov 02 17:43:59.794602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.794582 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:43:59.850523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.850492 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z"] Nov 02 17:43:59.873149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.873109 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" event={"ID":"7743d48e-e421-44ee-952c-97424e208e4c","Type":"ContainerStarted","Data":"dc5ffaa70a9ebb9ba84ebf047504a5190d08bad24a0fc55f57abb7b1026a8da1"} Nov 02 17:43:59.873330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.873170 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" event={"ID":"7743d48e-e421-44ee-952c-97424e208e4c","Type":"ContainerStarted","Data":"d0ec8966fcbdcb249e8a1ed16aa804dee0fef9ffdd06f4115772b572051b35f3"} Nov 02 17:43:59.917475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.917445 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbckp\" (UniqueName: \"kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:43:59.917654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.917491 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:43:59.917711 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:43:59.917688 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.018880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.018789 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.019055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.018883 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbckp\" (UniqueName: \"kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.019055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.018919 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.019565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.019545 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.021660 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.021633 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.024390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.024345 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" podStartSLOduration=2.024333127 podStartE2EDuration="2.024333127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:00.022032417 +0000 UTC m=+1227.095559622" watchObservedRunningTime="2026-11-02 17:44:00.024333127 +0000 UTC m=+1227.097860330" Nov 02 17:44:00.076697 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.076656 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbckp\" (UniqueName: \"kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp\") pod \"maas-api-revoke-sub-5aa62b97-2ht2z\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.107466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.107431 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:00.482104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.482072 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z"] Nov 02 17:44:00.482314 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:00.482293 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c08b62_6cd8_48f5_9ad3_da80790757ef.slice/crio-783c3f9c3865530a870eeee4174b7944456e9914499562640b1b9efa4a04cbb3 WatchSource:0}: Error finding container 783c3f9c3865530a870eeee4174b7944456e9914499562640b1b9efa4a04cbb3: Status 404 returned error can't find the container with id 783c3f9c3865530a870eeee4174b7944456e9914499562640b1b9efa4a04cbb3 Nov 02 17:44:00.881582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.881544 5647 generic.go:356] "Generic (PLEG): container finished" podID="7743d48e-e421-44ee-952c-97424e208e4c" containerID="dc5ffaa70a9ebb9ba84ebf047504a5190d08bad24a0fc55f57abb7b1026a8da1" exitCode=0 Nov 02 17:44:00.882030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.881633 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" event={"ID":"7743d48e-e421-44ee-952c-97424e208e4c","Type":"ContainerDied","Data":"dc5ffaa70a9ebb9ba84ebf047504a5190d08bad24a0fc55f57abb7b1026a8da1"} Nov 02 17:44:00.883368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.883345 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" event={"ID":"75c08b62-6cd8-48f5-9ad3-da80790757ef","Type":"ContainerStarted","Data":"35b150800f5295ad85816f541305f28c4cfce4de84175dc2d6d001ecdddbc79c"} Nov 02 17:44:00.883481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:00.883384 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" event={"ID":"75c08b62-6cd8-48f5-9ad3-da80790757ef","Type":"ContainerStarted","Data":"783c3f9c3865530a870eeee4174b7944456e9914499562640b1b9efa4a04cbb3"} Nov 02 17:44:01.127446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:01.127392 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" podStartSLOduration=2.127377714 podStartE2EDuration="2.127377714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:01.126701875 +0000 UTC m=+1228.200229079" watchObservedRunningTime="2026-11-02 17:44:01.127377714 +0000 UTC m=+1228.200904964" Nov 02 17:44:01.891725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:01.891688 5647 generic.go:356] "Generic (PLEG): container finished" podID="75c08b62-6cd8-48f5-9ad3-da80790757ef" containerID="35b150800f5295ad85816f541305f28c4cfce4de84175dc2d6d001ecdddbc79c" exitCode=0 Nov 02 17:44:01.892140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:01.891770 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" event={"ID":"75c08b62-6cd8-48f5-9ad3-da80790757ef","Type":"ContainerDied","Data":"35b150800f5295ad85816f541305f28c4cfce4de84175dc2d6d001ecdddbc79c"} Nov 02 17:44:02.049686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.049657 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:44:02.149838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.149688 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf\" (UniqueName: \"kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf\") pod \"7743d48e-e421-44ee-952c-97424e208e4c\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " Nov 02 17:44:02.149838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.149789 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca\") pod \"7743d48e-e421-44ee-952c-97424e208e4c\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " Nov 02 17:44:02.149838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.149833 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca\") pod \"7743d48e-e421-44ee-952c-97424e208e4c\" (UID: \"7743d48e-e421-44ee-952c-97424e208e4c\") " Nov 02 17:44:02.150174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.150143 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca" pod "7743d48e-e421-44ee-952c-97424e208e4c" (UID: "7743d48e-e421-44ee-952c-97424e208e4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:02.150371 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.150355 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:02.152031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.152005 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca" pod "7743d48e-e421-44ee-952c-97424e208e4c" (UID: "7743d48e-e421-44ee-952c-97424e208e4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:02.152148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.152132 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf" pod "7743d48e-e421-44ee-952c-97424e208e4c" (UID: "7743d48e-e421-44ee-952c-97424e208e4c"). InnerVolumeSpecName "kube-api-access-8dktf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:02.251878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.251842 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dktf\" (UniqueName: \"kubernetes.io/projected/7743d48e-e421-44ee-952c-97424e208e4c-kube-api-access-8dktf\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:02.251878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.251873 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7743d48e-e421-44ee-952c-97424e208e4c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:02.337473 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:02.337437 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:44:02.902767 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.902721 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:44:02.903345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.902742 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" event={"ID":"7743d48e-e421-44ee-952c-97424e208e4c","Type":"ContainerDied","Data":"d0ec8966fcbdcb249e8a1ed16aa804dee0fef9ffdd06f4115772b572051b35f3"} Nov 02 17:44:02.903345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.902868 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0ec8966fcbdcb249e8a1ed16aa804dee0fef9ffdd06f4115772b572051b35f3" Nov 02 17:44:02.973439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:02.973402 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:44:03.065875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.065845 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:03.162255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.162173 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp\" (UniqueName: \"kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp\") pod \"75c08b62-6cd8-48f5-9ad3-da80790757ef\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " Nov 02 17:44:03.162406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.162256 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca\") pod \"75c08b62-6cd8-48f5-9ad3-da80790757ef\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " Nov 02 17:44:03.162406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.162357 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca\") pod \"75c08b62-6cd8-48f5-9ad3-da80790757ef\" (UID: \"75c08b62-6cd8-48f5-9ad3-da80790757ef\") " Nov 02 17:44:03.162602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.162579 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca" pod "75c08b62-6cd8-48f5-9ad3-da80790757ef" (UID: "75c08b62-6cd8-48f5-9ad3-da80790757ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:03.162904 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.162884 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:03.164523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.164500 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca" pod "75c08b62-6cd8-48f5-9ad3-da80790757ef" (UID: "75c08b62-6cd8-48f5-9ad3-da80790757ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:03.164622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.164562 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp" pod "75c08b62-6cd8-48f5-9ad3-da80790757ef" (UID: "75c08b62-6cd8-48f5-9ad3-da80790757ef"). InnerVolumeSpecName "kube-api-access-zbckp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:03.264416 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.264376 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbckp\" (UniqueName: \"kubernetes.io/projected/75c08b62-6cd8-48f5-9ad3-da80790757ef-kube-api-access-zbckp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:03.264416 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.264414 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75c08b62-6cd8-48f5-9ad3-da80790757ef-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:03.933833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.933044 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" event={"ID":"75c08b62-6cd8-48f5-9ad3-da80790757ef","Type":"ContainerDied","Data":"783c3f9c3865530a870eeee4174b7944456e9914499562640b1b9efa4a04cbb3"} Nov 02 17:44:03.933833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.933124 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="783c3f9c3865530a870eeee4174b7944456e9914499562640b1b9efa4a04cbb3" Nov 02 17:44:03.933833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:03.933193 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z" Nov 02 17:44:06.214147 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:06.214101 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:44:06.259905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:06.259856 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.443958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:06.443895 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:06.599771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:06.598913 5647 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Nov 02 17:44:06.902923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:06.902819 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.233818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.233482 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm"] Nov 02 17:44:07.237802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.234401 5647 cpu_manager.go:414] "containerMap: removing container" podUID="7743d48e-e421-44ee-952c-97424e208e4c" containerName="revoke-keys" Nov 02 17:44:07.237802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.234430 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7743d48e-e421-44ee-952c-97424e208e4c" containerName="revoke-keys" Nov 02 17:44:07.237802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.234446 5647 cpu_manager.go:414] "containerMap: removing container" podUID="75c08b62-6cd8-48f5-9ad3-da80790757ef" containerName="revoke-keys" Nov 02 17:44:07.237802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.234454 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75c08b62-6cd8-48f5-9ad3-da80790757ef" containerName="revoke-keys" Nov 02 17:44:07.237802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.234633 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="7743d48e-e421-44ee-952c-97424e208e4c" containerName="revoke-keys" Nov 02 17:44:07.237802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.234648 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="75c08b62-6cd8-48f5-9ad3-da80790757ef" containerName="revoke-keys" Nov 02 17:44:07.283757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.273426 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 17:44:07.296754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.289386 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:44:07.296754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.291907 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335108 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335215 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335279 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335349 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335393 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335511 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335623 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzd4\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.344992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.335684 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.386360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.385417 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8cbedb9f-openshift-default-dockercfg-f6kqf\"" Nov 02 17:44:07.407002 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.406860 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm"] Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442272 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442371 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442434 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442481 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442522 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442633 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442681 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442757 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzd4\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.442801 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.453554 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.479874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.473633 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.506812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.481059 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 17:44:07.508402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.484862 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.508402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.484913 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:07.508402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.485176 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.508402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.499756 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.508402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.500024 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.508402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.507876 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:44:07.547214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.547157 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.790824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.780177 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzd4\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:07.826859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:07.816383 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:08.175500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.175455 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:08.339105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.337442 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 17:44:08.369611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.369331 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:08.522389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.509012 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:08.522389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.509312 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Nov 02 17:44:08.522389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.509882 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:44:08.552759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.548144 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded" Nov 02 17:44:08.552759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.549382 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:44:08.562323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.561155 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:44:08.611157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:08.611086 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:44:09.106823 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:09.090932 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:44:09.491043 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.491003 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:44:09.491557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.491075 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:44:09.491557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.491106 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm"] Nov 02 17:44:09.496651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.495906 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:09.496651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.496008 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:09.496651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.496050 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:09.634549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.634500 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq"] Nov 02 17:44:09.713348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.697498 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.714221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.714181 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq"] Nov 02 17:44:09.769298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.756782 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:44:09.769298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.756932 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:09.813185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.813147 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.813552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.813524 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sg5h\" (UniqueName: \"kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.813827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.813807 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.915052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.915015 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.915438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.915417 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sg5h\" (UniqueName: \"kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.916367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.916346 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.917022 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.915883 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.919419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.919374 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:09.974548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:09.974462 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sg5h\" (UniqueName: \"kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h\") pod \"maas-api-revoke-sub-c6c142a4-qbsmq\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:10.059644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:10.059608 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:10.443705 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:10.443667 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq"] Nov 02 17:44:10.443976 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:10.443948 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9101e4b5_2ab3_466d_bbcd_e238bb95e035.slice/crio-d547f2e2e935249d768fe5669edd43fb88c3df2b0bb0d226a9cc6bfb27681ad1 WatchSource:0}: Error finding container d547f2e2e935249d768fe5669edd43fb88c3df2b0bb0d226a9cc6bfb27681ad1: Status 404 returned error can't find the container with id d547f2e2e935249d768fe5669edd43fb88c3df2b0bb0d226a9cc6bfb27681ad1 Nov 02 17:44:10.505751 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:10.505690 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" event={"ID":"576113f5-04da-4871-980d-c101f55b1622","Type":"ContainerStarted","Data":"fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c"} Nov 02 17:44:10.506208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:10.505779 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" event={"ID":"576113f5-04da-4871-980d-c101f55b1622","Type":"ContainerStarted","Data":"917bd80fb52c3860d34420e5474f4167ba67b9917823b75960efa270bd2dcd2c"} Nov 02 17:44:10.507124 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:10.507094 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" event={"ID":"9101e4b5-2ab3-466d-bbcd-e238bb95e035","Type":"ContainerStarted","Data":"d547f2e2e935249d768fe5669edd43fb88c3df2b0bb0d226a9cc6bfb27681ad1"} Nov 02 17:44:10.627477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:10.627428 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podStartSLOduration=3.6274140470000003 podStartE2EDuration="3.627414047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:10.62616094 +0000 UTC m=+1237.699688144" watchObservedRunningTime="2026-11-02 17:44:10.627414047 +0000 UTC m=+1237.700941323" Nov 02 17:44:11.373904 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:11.373870 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:11.379136 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:11.379110 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:11.517454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:11.517425 5647 generic.go:356] "Generic (PLEG): container finished" podID="9101e4b5-2ab3-466d-bbcd-e238bb95e035" containerID="033402b0431dda235dde9dd2ff8575209a3cce09e9b677850cd428acdf880a50" exitCode=0 Nov 02 17:44:11.517839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:11.517511 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" event={"ID":"9101e4b5-2ab3-466d-bbcd-e238bb95e035","Type":"ContainerDied","Data":"033402b0431dda235dde9dd2ff8575209a3cce09e9b677850cd428acdf880a50"} Nov 02 17:44:11.518030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:11.518009 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:11.518977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:11.518952 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:44:12.673891 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.673862 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:12.865970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.865938 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca\") pod \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " Nov 02 17:44:12.866149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.866024 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h\" (UniqueName: \"kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h\") pod \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " Nov 02 17:44:12.866149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.866084 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca\") pod \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\" (UID: \"9101e4b5-2ab3-466d-bbcd-e238bb95e035\") " Nov 02 17:44:12.866311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.866286 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca" pod "9101e4b5-2ab3-466d-bbcd-e238bb95e035" (UID: "9101e4b5-2ab3-466d-bbcd-e238bb95e035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:12.866463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.866436 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:12.868310 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.868285 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca" pod "9101e4b5-2ab3-466d-bbcd-e238bb95e035" (UID: "9101e4b5-2ab3-466d-bbcd-e238bb95e035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:12.868396 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.868376 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h" pod "9101e4b5-2ab3-466d-bbcd-e238bb95e035" (UID: "9101e4b5-2ab3-466d-bbcd-e238bb95e035"). InnerVolumeSpecName "kube-api-access-2sg5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:12.967652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.967614 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sg5h\" (UniqueName: \"kubernetes.io/projected/9101e4b5-2ab3-466d-bbcd-e238bb95e035-kube-api-access-2sg5h\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:12.967652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:12.967647 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9101e4b5-2ab3-466d-bbcd-e238bb95e035-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:13.459626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.459595 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg"] Nov 02 17:44:13.460175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.460161 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9101e4b5-2ab3-466d-bbcd-e238bb95e035" containerName="revoke-keys" Nov 02 17:44:13.460251 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.460178 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9101e4b5-2ab3-466d-bbcd-e238bb95e035" containerName="revoke-keys" Nov 02 17:44:13.460291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.460280 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9101e4b5-2ab3-466d-bbcd-e238bb95e035" containerName="revoke-keys" Nov 02 17:44:13.463849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.463829 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.510414 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.510383 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd"] Nov 02 17:44:13.515280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.515260 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.537227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.537201 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" Nov 02 17:44:13.537403 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.537280 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq" event={"ID":"9101e4b5-2ab3-466d-bbcd-e238bb95e035","Type":"ContainerDied","Data":"d547f2e2e935249d768fe5669edd43fb88c3df2b0bb0d226a9cc6bfb27681ad1"} Nov 02 17:44:13.537403 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.537333 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d547f2e2e935249d768fe5669edd43fb88c3df2b0bb0d226a9cc6bfb27681ad1" Nov 02 17:44:13.569319 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.569190 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg"] Nov 02 17:44:13.575000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.574966 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575026 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575174 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575296 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575232 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575422 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575528 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575628 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575701 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575671 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.575771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.575728 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnchj\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.621095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.621060 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd"] Nov 02 17:44:13.677256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677202 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.677788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677294 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxctz\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.677788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677346 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.677788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677373 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.677788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677407 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.677788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677444 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.677788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.677505 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678096 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678147 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.678364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678222 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678261 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678297 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnchj\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678330 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678386 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.678541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678436 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.679191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678844 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.679191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678989 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.679372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.679338 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.679569 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.678477 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.679670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.679630 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.679773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.679745 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.679906 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.679852 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.679996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.679955 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.681615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.681590 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.682531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.682502 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.742308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.742235 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.742598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.742571 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnchj\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.778086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.778055 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:13.781331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781302 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781334 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781379 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781415 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781448 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxctz\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781485 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781560 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.781657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.781627 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.782034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.782011 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.782302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.782275 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.782302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.782294 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.782479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.782453 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.782651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.782629 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.784157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.784137 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.784427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.784408 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.842437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.842378 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:13.843271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:13.843248 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxctz\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:14.047862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.047776 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c"] Nov 02 17:44:14.051046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.051017 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69aaa5bf-hl85c"] Nov 02 17:44:14.129455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.129417 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:14.228539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.228505 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg"] Nov 02 17:44:14.229991 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:14.229952 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013678d9_8093_4f8d_ab06_cf5a7cfc46d0.slice/crio-faa0c2fa59613082046dab19342051579061a80352ed7b646b4c2142c2f8c458 WatchSource:0}: Error finding container faa0c2fa59613082046dab19342051579061a80352ed7b646b4c2142c2f8c458: Status 404 returned error can't find the container with id faa0c2fa59613082046dab19342051579061a80352ed7b646b4c2142c2f8c458 Nov 02 17:44:14.232821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.232786 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:14.232920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.232853 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:14.232920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.232884 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:14.515661 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.515631 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd"] Nov 02 17:44:14.517419 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:14.517384 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0bdf34_2fb7_4820_a7d9_75ce6c57b956.slice/crio-781828897a18d298c3f0f868c91888bf4a3f7811eba574feed5cce2f9a985047 WatchSource:0}: Error finding container 781828897a18d298c3f0f868c91888bf4a3f7811eba574feed5cce2f9a985047: Status 404 returned error can't find the container with id 781828897a18d298c3f0f868c91888bf4a3f7811eba574feed5cce2f9a985047 Nov 02 17:44:14.520931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.520893 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:14.521005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.520981 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:14.521059 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.521028 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:14.549351 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.549317 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" event={"ID":"013678d9-8093-4f8d-ab06-cf5a7cfc46d0","Type":"ContainerStarted","Data":"9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac"} Nov 02 17:44:14.549524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.549378 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" event={"ID":"013678d9-8093-4f8d-ab06-cf5a7cfc46d0","Type":"ContainerStarted","Data":"faa0c2fa59613082046dab19342051579061a80352ed7b646b4c2142c2f8c458"} Nov 02 17:44:14.551035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.551012 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" event={"ID":"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956","Type":"ContainerStarted","Data":"781828897a18d298c3f0f868c91888bf4a3f7811eba574feed5cce2f9a985047"} Nov 02 17:44:14.672058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.671989 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podStartSLOduration=1.6719678409999998 podStartE2EDuration="1.671967841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:14.667600128 +0000 UTC m=+1241.741127332" watchObservedRunningTime="2026-11-02 17:44:14.671967841 +0000 UTC m=+1241.745495021" Nov 02 17:44:14.780752 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:14.780622 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:15.086717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:15.086685 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed97dbd4-d222-4d18-8250-c9055487ae40" path="/var/lib/kubelet/pods/ed97dbd4-d222-4d18-8250-c9055487ae40/volumes" Nov 02 17:44:15.566718 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:15.566059 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" event={"ID":"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956","Type":"ContainerStarted","Data":"c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b"} Nov 02 17:44:15.780446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:15.780411 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:44:15.780603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:15.780475 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 17:44:15.790530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:15.790483 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podStartSLOduration=2.790469492 podStartE2EDuration="2.790469492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:15.739203571 +0000 UTC m=+1242.812730774" watchObservedRunningTime="2026-11-02 17:44:15.790469492 +0000 UTC m=+1242.863996695" Nov 02 17:44:16.135486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:16.135448 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:16.135923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:16.135894 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:44:16.136159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:16.136130 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:44:16.778498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:16.778459 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:44:16.778686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:16.778508 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 17:44:17.138501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:17.138454 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:44:17.139177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:17.139141 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:44:17.817151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:17.796162 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:44:17.817151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:17.796221 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 17:44:17.974002 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:17.973769 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:44:18.134977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.134941 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:44:18.135293 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.135262 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:44:18.779254 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.779216 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:44:18.779801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.779272 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 17:44:18.842192 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.842153 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424"] Nov 02 17:44:18.855214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.855183 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:18.872236 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.872208 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424"] Nov 02 17:44:18.937769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.936014 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:44:18.937769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.936071 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:18.962702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.962659 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:18.962927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.962848 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:18.963251 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:18.963225 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6n4\" (UniqueName: \"kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.064331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.064243 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6n4\" (UniqueName: \"kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.064331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.064300 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.064529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.064363 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.065013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.064996 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.067637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.067618 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.133761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.129870 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:44:19.168765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.134042 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:44:19.175760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.175526 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6n4\" (UniqueName: \"kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4\") pod \"maas-api-revoke-sub-5ef26c8b-jr424\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.183782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.176513 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:19.247255 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:19.233659 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:44:19.779342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.779312 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:44:19.779763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.779362 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 17:44:19.865572 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:19.865205 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424"] Nov 02 17:44:19.866449 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:19.866413 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd7c4aa_9171_4a18_9eb1_209b1572a861.slice/crio-c87864e7f98d20cf9dc5a5e80d6cc4d3005a79a0f6101249b92c357a3396621b WatchSource:0}: Error finding container c87864e7f98d20cf9dc5a5e80d6cc4d3005a79a0f6101249b92c357a3396621b: Status 404 returned error can't find the container with id c87864e7f98d20cf9dc5a5e80d6cc4d3005a79a0f6101249b92c357a3396621b Nov 02 17:44:20.143759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.132273 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:44:20.143759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.132334 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:44:20.779062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.779025 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 02 17:44:20.779264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.779079 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 02 17:44:20.877843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.877806 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" event={"ID":"bfd7c4aa-9171-4a18-9eb1-209b1572a861","Type":"ContainerStarted","Data":"58f008e8111764ac8ac55d36b5342fb1a1af82faec4ae705f4107342b9ab7829"} Nov 02 17:44:20.878273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.877865 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" event={"ID":"bfd7c4aa-9171-4a18-9eb1-209b1572a861","Type":"ContainerStarted","Data":"c87864e7f98d20cf9dc5a5e80d6cc4d3005a79a0f6101249b92c357a3396621b"} Nov 02 17:44:20.996317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:20.996267 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" podStartSLOduration=2.996253282 podStartE2EDuration="2.996253282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:20.99456576 +0000 UTC m=+1248.068092963" watchObservedRunningTime="2026-11-02 17:44:20.996253282 +0000 UTC m=+1248.069780485" Nov 02 17:44:21.130379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:21.130344 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Nov 02 17:44:21.130559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:21.130414 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Nov 02 17:44:21.889214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:21.889172 5647 generic.go:356] "Generic (PLEG): container finished" podID="bfd7c4aa-9171-4a18-9eb1-209b1572a861" containerID="58f008e8111764ac8ac55d36b5342fb1a1af82faec4ae705f4107342b9ab7829" exitCode=0 Nov 02 17:44:21.889638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:21.889256 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" event={"ID":"bfd7c4aa-9171-4a18-9eb1-209b1572a861","Type":"ContainerDied","Data":"58f008e8111764ac8ac55d36b5342fb1a1af82faec4ae705f4107342b9ab7829"} Nov 02 17:44:22.281440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.281351 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:22.281790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.281767 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:22.282901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.282841 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:44:22.586015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.585987 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:22.586574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.586551 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:22.587596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.587576 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:44:22.976458 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.976425 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr"] Nov 02 17:44:22.984659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:22.984628 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024163 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024125 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024173 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024279 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024321 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22hjp\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024458 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024515 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024506 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024583 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024767 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.024863 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.024799 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.041355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.041322 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr"] Nov 02 17:44:23.074343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.074312 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:23.125806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.125777 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca\") pod \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " Nov 02 17:44:23.125968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.125917 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca\") pod \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " Nov 02 17:44:23.125968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.125955 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4\" (UniqueName: \"kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4\") pod \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\" (UID: \"bfd7c4aa-9171-4a18-9eb1-209b1572a861\") " Nov 02 17:44:23.126131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126115 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126143 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126140 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca" pod "bfd7c4aa-9171-4a18-9eb1-209b1572a861" (UID: "bfd7c4aa-9171-4a18-9eb1-209b1572a861"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:23.126266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126248 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126313 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126276 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126313 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126307 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126331 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22hjp\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126546 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126525 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126596 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126648 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126651 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.126858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126841 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:23.126964 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.126945 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.127266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.127240 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.127453 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.127427 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.127453 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.127444 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.128641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.128618 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.128857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.128831 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca" pod "bfd7c4aa-9171-4a18-9eb1-209b1572a861" (UID: "bfd7c4aa-9171-4a18-9eb1-209b1572a861"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:23.128951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.128868 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4" pod "bfd7c4aa-9171-4a18-9eb1-209b1572a861" (UID: "bfd7c4aa-9171-4a18-9eb1-209b1572a861"). InnerVolumeSpecName "kube-api-access-sq6n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:23.129093 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.129076 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.184248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.184182 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.184382 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.184336 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22hjp\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.228119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.228084 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd7c4aa-9171-4a18-9eb1-209b1572a861-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:23.228302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.228120 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6n4\" (UniqueName: \"kubernetes.io/projected/bfd7c4aa-9171-4a18-9eb1-209b1572a861-kube-api-access-sq6n4\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:23.306829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.306800 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:23.755304 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.755271 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr"] Nov 02 17:44:23.755421 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:23.755334 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116d27ce_62ce_42cc_a12d_386da00fef75.slice/crio-44638c6e30d76a40c29da51e8b7b47cf27b4c9bb169e0f520bb44cd5fc1346af WatchSource:0}: Error finding container 44638c6e30d76a40c29da51e8b7b47cf27b4c9bb169e0f520bb44cd5fc1346af: Status 404 returned error can't find the container with id 44638c6e30d76a40c29da51e8b7b47cf27b4c9bb169e0f520bb44cd5fc1346af Nov 02 17:44:23.757692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.757660 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:23.757802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.757743 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:23.757802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.757775 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:23.910804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.910767 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" event={"ID":"bfd7c4aa-9171-4a18-9eb1-209b1572a861","Type":"ContainerDied","Data":"c87864e7f98d20cf9dc5a5e80d6cc4d3005a79a0f6101249b92c357a3396621b"} Nov 02 17:44:23.910804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.910802 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424" Nov 02 17:44:23.911060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.910829 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87864e7f98d20cf9dc5a5e80d6cc4d3005a79a0f6101249b92c357a3396621b" Nov 02 17:44:23.913176 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.913113 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" event={"ID":"116d27ce-62ce-42cc-a12d-386da00fef75","Type":"ContainerStarted","Data":"36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3"} Nov 02 17:44:23.913560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:23.913361 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" event={"ID":"116d27ce-62ce-42cc-a12d-386da00fef75","Type":"ContainerStarted","Data":"44638c6e30d76a40c29da51e8b7b47cf27b4c9bb169e0f520bb44cd5fc1346af"} Nov 02 17:44:24.034115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:24.033993 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podStartSLOduration=2.033970698 podStartE2EDuration="2.033970698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:24.032458489 +0000 UTC m=+1251.105985717" watchObservedRunningTime="2026-11-02 17:44:24.033970698 +0000 UTC m=+1251.107497904" Nov 02 17:44:24.307813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:24.307701 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:24.309156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:24.309131 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:24.309290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:24.309176 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:25.315216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.315170 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:25.319965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.319907 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:25.629330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.629277 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg"] Nov 02 17:44:25.630112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.630083 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bfd7c4aa-9171-4a18-9eb1-209b1572a861" containerName="revoke-keys" Nov 02 17:44:25.630112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.630107 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd7c4aa-9171-4a18-9eb1-209b1572a861" containerName="revoke-keys" Nov 02 17:44:25.630316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.630301 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd7c4aa-9171-4a18-9eb1-209b1572a861" containerName="revoke-keys" Nov 02 17:44:25.634052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.634032 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.664237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.664202 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.664392 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.664344 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.664477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.664464 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfzz\" (UniqueName: \"kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.671576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.671540 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg"] Nov 02 17:44:25.727762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.727061 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:44:25.727762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.727188 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:25.765381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.765345 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfzz\" (UniqueName: \"kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.765592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.765459 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.765592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.765515 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.766162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.766133 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.768128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.768108 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.829038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.828999 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfzz\" (UniqueName: \"kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz\") pod \"maas-api-revoke-sub-07d6463a-5t7fg\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:25.948499 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:25.948419 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:26.315823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:26.311131 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:26.315823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:26.311195 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:26.429746 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:26.429704 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg"] Nov 02 17:44:26.432473 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:26.432443 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda012f974_c290_4733_af00_1199d668e0ef.slice/crio-bc981399edf73e3a5c95a7fe98d537bf1606f09327ef1dfc3cbd9e518b6e0bce WatchSource:0}: Error finding container bc981399edf73e3a5c95a7fe98d537bf1606f09327ef1dfc3cbd9e518b6e0bce: Status 404 returned error can't find the container with id bc981399edf73e3a5c95a7fe98d537bf1606f09327ef1dfc3cbd9e518b6e0bce Nov 02 17:44:26.951134 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:26.951032 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" event={"ID":"a012f974-c290-4733-af00-1199d668e0ef","Type":"ContainerStarted","Data":"319f150a779682161d9090b46d0e27b1e0d077bbf8904f4ca0012e8dfe291dec"} Nov 02 17:44:26.951134 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:26.951105 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" event={"ID":"a012f974-c290-4733-af00-1199d668e0ef","Type":"ContainerStarted","Data":"bc981399edf73e3a5c95a7fe98d537bf1606f09327ef1dfc3cbd9e518b6e0bce"} Nov 02 17:44:27.089243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:27.089177 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" podStartSLOduration=2.089161348 podStartE2EDuration="2.089161348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:27.08877825 +0000 UTC m=+1254.162305453" watchObservedRunningTime="2026-11-02 17:44:27.089161348 +0000 UTC m=+1254.162688551" Nov 02 17:44:27.319267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:27.319178 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:27.319864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:27.319823 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:27.960149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:27.960110 5647 generic.go:356] "Generic (PLEG): container finished" podID="a012f974-c290-4733-af00-1199d668e0ef" containerID="319f150a779682161d9090b46d0e27b1e0d077bbf8904f4ca0012e8dfe291dec" exitCode=0 Nov 02 17:44:27.960361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:27.960198 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" event={"ID":"a012f974-c290-4733-af00-1199d668e0ef","Type":"ContainerDied","Data":"319f150a779682161d9090b46d0e27b1e0d077bbf8904f4ca0012e8dfe291dec"} Nov 02 17:44:28.311297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:28.311195 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:28.311573 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:28.311542 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:29.169204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.169174 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:29.236388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.236298 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz\" (UniqueName: \"kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz\") pod \"a012f974-c290-4733-af00-1199d668e0ef\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " Nov 02 17:44:29.236552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.236500 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca\") pod \"a012f974-c290-4733-af00-1199d668e0ef\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " Nov 02 17:44:29.236552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.236544 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca\") pod \"a012f974-c290-4733-af00-1199d668e0ef\" (UID: \"a012f974-c290-4733-af00-1199d668e0ef\") " Nov 02 17:44:29.236862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.236826 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca" pod "a012f974-c290-4733-af00-1199d668e0ef" (UID: "a012f974-c290-4733-af00-1199d668e0ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:29.237354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.237334 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a012f974-c290-4733-af00-1199d668e0ef-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:29.238685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.238662 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz" pod "a012f974-c290-4733-af00-1199d668e0ef" (UID: "a012f974-c290-4733-af00-1199d668e0ef"). InnerVolumeSpecName "kube-api-access-zwfzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:29.238935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.238918 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca" pod "a012f974-c290-4733-af00-1199d668e0ef" (UID: "a012f974-c290-4733-af00-1199d668e0ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:29.326905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.326862 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:29.327212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.327172 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:29.338481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.338449 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a012f974-c290-4733-af00-1199d668e0ef-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:29.338775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.338754 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwfzz\" (UniqueName: \"kubernetes.io/projected/a012f974-c290-4733-af00-1199d668e0ef-kube-api-access-zwfzz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:29.402499 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:29.402453 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache]" Nov 02 17:44:29.984858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.984831 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" Nov 02 17:44:29.985933 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.985097 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg" event={"ID":"a012f974-c290-4733-af00-1199d668e0ef","Type":"ContainerDied","Data":"bc981399edf73e3a5c95a7fe98d537bf1606f09327ef1dfc3cbd9e518b6e0bce"} Nov 02 17:44:29.986052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:29.985998 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc981399edf73e3a5c95a7fe98d537bf1606f09327ef1dfc3cbd9e518b6e0bce" Nov 02 17:44:30.307542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:30.307443 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Nov 02 17:44:30.307542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:30.307514 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Nov 02 17:44:32.315760 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:32.298336 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1116c286_a714_43f3_bc7a_99e8e19d7cfe.slice/crio-ff091476d10c4c721481e05379f3d5966888300ba1380804d7f469b52dfd6ee0\": RecentStats: unable to find data in memory cache]" Nov 02 17:44:32.379211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:32.379170 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:32.379515 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:32.379479 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:33.402323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.402256 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 17:44:33.453962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.453919 5647 scope.go:122] "RemoveContainer" containerID="bf275f33e0290f9645cf50d7f01393a8ce25e1563874ecf53b42a238d90aa68f" Nov 02 17:44:33.464066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.462841 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:33.464066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.462930 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:33.520785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.516318 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:33.563765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.562955 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:44:33.563765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.563051 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:33.622770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.621851 5647 scope.go:122] "RemoveContainer" containerID="b1ef8dc1657d9b3435aad802b8c102525e2e32ea1af7cd8b627c8bf4d1e36127" Nov 02 17:44:33.704284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.704203 5647 scope.go:122] "RemoveContainer" containerID="2624d1332e9485c603180dbce2be15d1121294a8f9702e47d29d5ef9db1603c9" Nov 02 17:44:33.897288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.897258 5647 scope.go:122] "RemoveContainer" containerID="13988390fdc0333d763f7f275c343e75df85273560cd3723bcb796e09228da29" Nov 02 17:44:33.912247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.912198 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:33.983228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.983086 5647 scope.go:122] "RemoveContainer" containerID="b1283ca937d38853f54ef30547919b76138e8cc8d562891843098c4500b6e674" Nov 02 17:44:33.991870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:33.985843 5647 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:44:34.363605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:34.353718 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:34.363605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:34.355132 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:34.363605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:34.357970 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:44:35.948534 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:35.948494 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:44:38.096828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.096793 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp"] Nov 02 17:44:38.097352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.097339 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a012f974-c290-4733-af00-1199d668e0ef" containerName="revoke-keys" Nov 02 17:44:38.097403 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.097355 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a012f974-c290-4733-af00-1199d668e0ef" containerName="revoke-keys" Nov 02 17:44:38.097470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.097462 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a012f974-c290-4733-af00-1199d668e0ef" containerName="revoke-keys" Nov 02 17:44:38.101109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.101091 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.149988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.149953 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.150178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.150026 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.150178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.150134 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.150311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.150212 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.150370 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.150356 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.150529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.150492 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.150679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.150656 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.151087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.151053 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvht\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.151511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.151146 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.210617 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.210584 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp"] Nov 02 17:44:38.253348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.253308 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.254058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.254031 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.256976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.256950 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.257528 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.257500 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.257704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.257684 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.258026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.257954 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.258369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.258217 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.258778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.258726 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.259038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.259009 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvht\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.260137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.257429 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.260331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.256798 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.260438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.260389 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.261206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.261181 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.261387 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.261367 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.296559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.294692 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.301760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.301028 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.313430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.313360 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.328512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.328442 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvht\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.415687 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.415330 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:38.906237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.905898 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp"] Nov 02 17:44:38.951790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.947945 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6"] Nov 02 17:44:38.958183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.957573 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:38.958183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.957690 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:38.958183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.957755 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:38.966976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:38.965125 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.007308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.007267 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6"] Nov 02 17:44:39.022475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.022083 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:44:39.022744 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.022697 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:44:39.093789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.093722 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.093940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.093903 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.094043 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.094022 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7zkv\" (UniqueName: \"kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.207811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.203627 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.207811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.204396 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.207811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.207630 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.207811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.207775 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7zkv\" (UniqueName: \"kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.224854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.224639 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.270879 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.270846 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7zkv\" (UniqueName: \"kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv\") pod \"maas-api-revoke-sub-ebce122c-fktj6\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.286749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.286690 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:39.482266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.482178 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" event={"ID":"13b8c1b5-e126-4171-aba8-6a6f970327d6","Type":"ContainerStarted","Data":"40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e"} Nov 02 17:44:39.482266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.482230 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" event={"ID":"13b8c1b5-e126-4171-aba8-6a6f970327d6","Type":"ContainerStarted","Data":"7f4dd8ba02f13bb67f6b377a8d8b9450025c2cb630d04f58124918ef0c2010eb"} Nov 02 17:44:39.610637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.610584 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podStartSLOduration=1.610557253 podStartE2EDuration="1.610557253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:39.608415816 +0000 UTC m=+1266.681943020" watchObservedRunningTime="2026-11-02 17:44:39.610557253 +0000 UTC m=+1266.684084455" Nov 02 17:44:39.722924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.722890 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6"] Nov 02 17:44:39.723082 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:39.722983 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3cab67_aa08_467b_9c77_e6cfd96ee9c3.slice/crio-36b1ba0fb3a8988b22e85d4670d899b5eef98679f6ab954a08772b6f7e26abab WatchSource:0}: Error finding container 36b1ba0fb3a8988b22e85d4670d899b5eef98679f6ab954a08772b6f7e26abab: Status 404 returned error can't find the container with id 36b1ba0fb3a8988b22e85d4670d899b5eef98679f6ab954a08772b6f7e26abab Nov 02 17:44:39.801135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.801097 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6"] Nov 02 17:44:39.806014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.805985 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:39.862191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.862157 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6"] Nov 02 17:44:39.917803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.917766 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:39.917803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.917806 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jktjj\" (UniqueName: \"kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:39.918053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:39.918013 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.030018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.026955 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.030018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.027170 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.030018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.027222 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jktjj\" (UniqueName: \"kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.030018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.028596 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.033393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.033359 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.113829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.111488 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jktjj\" (UniqueName: \"kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj\") pod \"maas-api-revoke-sub-8c21d131-g64b6\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.123762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.122998 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:40.425643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.425196 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:40.436059 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.436022 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:40.436222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.436081 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:40.490870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.490833 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" event={"ID":"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3","Type":"ContainerStarted","Data":"7f9e909be187933442ed3db0bb8f246b4ccabee4fff6c940aa697534c4eab3cf"} Nov 02 17:44:40.491050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.490887 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" event={"ID":"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3","Type":"ContainerStarted","Data":"36b1ba0fb3a8988b22e85d4670d899b5eef98679f6ab954a08772b6f7e26abab"} Nov 02 17:44:40.618870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.618814 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" podStartSLOduration=2.618799127 podStartE2EDuration="2.618799127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:40.618169027 +0000 UTC m=+1267.691696231" watchObservedRunningTime="2026-11-02 17:44:40.618799127 +0000 UTC m=+1267.692326333" Nov 02 17:44:40.736256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:40.735556 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6"] Nov 02 17:44:41.425866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.424870 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:41.425866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.424935 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:41.499978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.499941 5647 generic.go:356] "Generic (PLEG): container finished" podID="8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" containerID="7f9e909be187933442ed3db0bb8f246b4ccabee4fff6c940aa697534c4eab3cf" exitCode=0 Nov 02 17:44:41.500164 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.500017 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" event={"ID":"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3","Type":"ContainerDied","Data":"7f9e909be187933442ed3db0bb8f246b4ccabee4fff6c940aa697534c4eab3cf"} Nov 02 17:44:41.501618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.501591 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" event={"ID":"966e4d2f-6350-47b4-942c-8b66bc87fe3f","Type":"ContainerStarted","Data":"769a4b4c5b5c854854d33649a27156ea12fcb0c1948e2d411897e25392e0d437"} Nov 02 17:44:41.501790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.501638 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" event={"ID":"966e4d2f-6350-47b4-942c-8b66bc87fe3f","Type":"ContainerStarted","Data":"d9cc8e6e810712fd05ddbbf4c9ae7b966a7b785663533a701ab6ed8329244da6"} Nov 02 17:44:41.745056 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:41.744952 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" podStartSLOduration=2.744934507 podStartE2EDuration="2.744934507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:41.741347756 +0000 UTC m=+1268.814874950" watchObservedRunningTime="2026-11-02 17:44:41.744934507 +0000 UTC m=+1268.818461709" Nov 02 17:44:42.425761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.419140 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:42.425761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.419239 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:42.543050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.543015 5647 generic.go:356] "Generic (PLEG): container finished" podID="966e4d2f-6350-47b4-942c-8b66bc87fe3f" containerID="769a4b4c5b5c854854d33649a27156ea12fcb0c1948e2d411897e25392e0d437" exitCode=0 Nov 02 17:44:42.543224 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.543080 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" event={"ID":"966e4d2f-6350-47b4-942c-8b66bc87fe3f","Type":"ContainerDied","Data":"769a4b4c5b5c854854d33649a27156ea12fcb0c1948e2d411897e25392e0d437"} Nov 02 17:44:42.741500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.741471 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:42.773842 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.773813 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv\" (UniqueName: \"kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv\") pod \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " Nov 02 17:44:42.774005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.773986 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca\") pod \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " Nov 02 17:44:42.774042 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.774021 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca\") pod \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\" (UID: \"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3\") " Nov 02 17:44:42.774345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.774320 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca" pod "8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" (UID: "8a3cab67-aa08-467b-9c77-e6cfd96ee9c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:42.774529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.774510 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:42.776071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.776050 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv" pod "8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" (UID: "8a3cab67-aa08-467b-9c77-e6cfd96ee9c3"). InnerVolumeSpecName "kube-api-access-j7zkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:42.776208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.776196 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca" pod "8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" (UID: "8a3cab67-aa08-467b-9c77-e6cfd96ee9c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:42.876320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.876282 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7zkv\" (UniqueName: \"kubernetes.io/projected/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-kube-api-access-j7zkv\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:42.876320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:42.876317 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:43.429843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.429697 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:43.430413 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.430380 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:43.551982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.551953 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" Nov 02 17:44:43.552171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.551978 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6" event={"ID":"8a3cab67-aa08-467b-9c77-e6cfd96ee9c3","Type":"ContainerDied","Data":"36b1ba0fb3a8988b22e85d4670d899b5eef98679f6ab954a08772b6f7e26abab"} Nov 02 17:44:43.552171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.552061 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36b1ba0fb3a8988b22e85d4670d899b5eef98679f6ab954a08772b6f7e26abab" Nov 02 17:44:43.775691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.775667 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:43.890692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.890657 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca\") pod \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " Nov 02 17:44:43.890902 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.890774 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj\" (UniqueName: \"kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj\") pod \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " Nov 02 17:44:43.890983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.890947 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca\") pod \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\" (UID: \"966e4d2f-6350-47b4-942c-8b66bc87fe3f\") " Nov 02 17:44:43.891116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.891088 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca" pod "966e4d2f-6350-47b4-942c-8b66bc87fe3f" (UID: "966e4d2f-6350-47b4-942c-8b66bc87fe3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:43.891636 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.891613 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:43.893775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.893678 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj" pod "966e4d2f-6350-47b4-942c-8b66bc87fe3f" (UID: "966e4d2f-6350-47b4-942c-8b66bc87fe3f"). InnerVolumeSpecName "kube-api-access-jktjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:43.894184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:43.894147 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca" pod "966e4d2f-6350-47b4-942c-8b66bc87fe3f" (UID: "966e4d2f-6350-47b4-942c-8b66bc87fe3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:44:44.096582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.077055 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966e4d2f-6350-47b4-942c-8b66bc87fe3f-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:44.113761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.105840 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jktjj\" (UniqueName: \"kubernetes.io/projected/966e4d2f-6350-47b4-942c-8b66bc87fe3f-kube-api-access-jktjj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:44.426839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.421186 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:44.426839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.421269 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:44.560239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.560200 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" Nov 02 17:44:44.560669 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.560251 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6" event={"ID":"966e4d2f-6350-47b4-942c-8b66bc87fe3f","Type":"ContainerDied","Data":"d9cc8e6e810712fd05ddbbf4c9ae7b966a7b785663533a701ab6ed8329244da6"} Nov 02 17:44:44.560669 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:44.560298 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9cc8e6e810712fd05ddbbf4c9ae7b966a7b785663533a701ab6ed8329244da6" Nov 02 17:44:45.419760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:45.417353 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:45.419760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:45.417415 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:46.416158 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:46.416121 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 02 17:44:46.416779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:46.416725 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 02 17:44:47.420719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:47.420686 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:47.421171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:47.421151 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:47.421803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:47.421784 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:44:50.233903 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.233871 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7981791ff29e: secret "agc-cred-7981791ff29e" not found Nov 02 17:44:50.234523 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.234237 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4530acdcb45: secret "agc-cred-d4530acdcb45" not found Nov 02 17:44:50.234523 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.234262 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq: [secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:50.234523 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.234331 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials podName:dc162bf2-1da3-4a5e-880b-3f9230e5e2ec nodeName:}" failed. No retries permitted until 2026-11-02 17:44:50.734310946 +0000 UTC m=+1277.807838156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials") pod "payload-processing-external-model-7599585dbd-lsvnq" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec") : [secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:50.741304 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.741271 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7981791ff29e: secret "agc-cred-7981791ff29e" not found Nov 02 17:44:50.741304 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.741300 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4530acdcb45: secret "agc-cred-d4530acdcb45" not found Nov 02 17:44:50.741304 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.741309 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq: [secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:50.741560 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:50.741370 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials podName:dc162bf2-1da3-4a5e-880b-3f9230e5e2ec nodeName:}" failed. No retries permitted until 2026-11-02 17:44:51.741355033 +0000 UTC m=+1278.814882217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials") pod "payload-processing-external-model-7599585dbd-lsvnq" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec") : [secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:51.720034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.719995 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4"] Nov 02 17:44:51.722712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.722647 5647 cpu_manager.go:414] "containerMap: removing container" podUID="8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" containerName="revoke-keys" Nov 02 17:44:51.722712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.722705 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" containerName="revoke-keys" Nov 02 17:44:51.722970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.722802 5647 cpu_manager.go:414] "containerMap: removing container" podUID="966e4d2f-6350-47b4-942c-8b66bc87fe3f" containerName="revoke-keys" Nov 02 17:44:51.722970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.722811 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966e4d2f-6350-47b4-942c-8b66bc87fe3f" containerName="revoke-keys" Nov 02 17:44:51.723309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.723282 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="966e4d2f-6350-47b4-942c-8b66bc87fe3f" containerName="revoke-keys" Nov 02 17:44:51.723309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.723310 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" containerName="revoke-keys" Nov 02 17:44:51.758135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.758096 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.890760 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:51.792660 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7981791ff29e: secret "agc-cred-7981791ff29e" not found Nov 02 17:44:51.985888 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:51.985804 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4530acdcb45: secret "agc-cred-d4530acdcb45" not found Nov 02 17:44:51.986467 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:51.986434 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq: [secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:51.986793 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:51.986774 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials podName:dc162bf2-1da3-4a5e-880b-3f9230e5e2ec nodeName:}" failed. No retries permitted until 2026-11-02 17:44:53.986746576 +0000 UTC m=+1281.060273773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials") pod "payload-processing-external-model-7599585dbd-lsvnq" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec") : [secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:51.987370 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.892239 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f8d61e-openshift-default-dockercfg-7qqpg\"" Nov 02 17:44:51.988476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.922965 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.988952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.988928 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.989162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.989145 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.989335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.989317 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.989547 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.989528 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tg9x\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.989750 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.989716 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.989975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.989954 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.990147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.990131 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.990324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.990307 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:51.990679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:51.986391 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4"] Nov 02 17:44:52.124311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.123448 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.124745 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.124705 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.126666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.126641 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.126857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.126838 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.127062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.127043 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tg9x\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.127228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.127210 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.127413 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.127395 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.127549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.127533 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.127692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.127673 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.128184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.128164 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.128314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.128201 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.128420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.127210 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.128509 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.128424 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.128593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.126793 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.129153 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.129095 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.132375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.132352 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.187978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.187919 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tg9x\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.188129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.188103 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token\") pod \"e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.406759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.403931 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:52.984611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.984575 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq"] Nov 02 17:44:52.991448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:52.991421 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4"] Nov 02 17:44:53.035032 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:44:53.034997 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430deb5a_0705_48e0_a500_2fa09af8a42d.slice/crio-f00e43c9a1c506c721f422774c7caa6e75471192d53ced8e32de86590f641abd WatchSource:0}: Error finding container f00e43c9a1c506c721f422774c7caa6e75471192d53ced8e32de86590f641abd: Status 404 returned error can't find the container with id f00e43c9a1c506c721f422774c7caa6e75471192d53ced8e32de86590f641abd Nov 02 17:44:53.037576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:53.037534 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:53.037649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:53.037629 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:53.037692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:53.037679 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:44:53.696757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:53.696028 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" event={"ID":"430deb5a-0705-48e0-a500-2fa09af8a42d","Type":"ContainerStarted","Data":"02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826"} Nov 02 17:44:53.696757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:53.696109 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" event={"ID":"430deb5a-0705-48e0-a500-2fa09af8a42d","Type":"ContainerStarted","Data":"f00e43c9a1c506c721f422774c7caa6e75471192d53ced8e32de86590f641abd"} Nov 02 17:44:54.088816 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:54.082711 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2296d33cdbd2: secret "agc-cred-2296d33cdbd2" not found Nov 02 17:44:54.088816 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:54.082838 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7981791ff29e: secret "agc-cred-7981791ff29e" not found Nov 02 17:44:54.088816 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:54.082849 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4530acdcb45: secret "agc-cred-d4530acdcb45" not found Nov 02 17:44:54.088816 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:54.082864 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq: [secret "agc-cred-2296d33cdbd2" not found, secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:54.088816 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:54.082943 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials podName:dc162bf2-1da3-4a5e-880b-3f9230e5e2ec nodeName:}" failed. No retries permitted until 2026-11-02 17:44:58.082925949 +0000 UTC m=+1285.156453143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials") pod "payload-processing-external-model-7599585dbd-lsvnq" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec") : [secret "agc-cred-2296d33cdbd2" not found, secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:54.408095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:54.407958 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:54.423673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:54.423196 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:54.542365 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:54.542307 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podStartSLOduration=3.542293372 podStartE2EDuration="3.542293372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:53.844312488 +0000 UTC m=+1280.917839700" watchObservedRunningTime="2026-11-02 17:44:54.542293372 +0000 UTC m=+1281.615820575" Nov 02 17:44:54.703975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:54.703883 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:54.704688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:54.704671 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:44:57.985613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:57.985557 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" podUID="dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" containerName="payload-processing" containerID="cri-o://88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9" gracePeriod=25 Nov 02 17:44:58.144237 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:58.144204 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-2296d33cdbd2: secret "agc-cred-2296d33cdbd2" not found Nov 02 17:44:58.144237 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:58.144234 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7981791ff29e: secret "agc-cred-7981791ff29e" not found Nov 02 17:44:58.144237 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:58.144241 5647 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4530acdcb45: secret "agc-cred-d4530acdcb45" not found Nov 02 17:44:58.144471 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:58.144251 5647 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq: [secret "agc-cred-2296d33cdbd2" not found, secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:58.144471 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:58.144323 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials podName:dc162bf2-1da3-4a5e-880b-3f9230e5e2ec nodeName:}" failed. No retries permitted until 2026-11-02 17:45:06.144306818 +0000 UTC m=+1293.217834012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials") pod "payload-processing-external-model-7599585dbd-lsvnq" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec") : [secret "agc-cred-2296d33cdbd2" not found, secret "agc-cred-7981791ff29e" not found, secret "agc-cred-d4530acdcb45" not found] Nov 02 17:44:58.255671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.255596 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:44:58.346581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.346550 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials\") pod \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " Nov 02 17:44:58.346581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.346593 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2\") pod \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " Nov 02 17:44:58.346858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.346681 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume\") pod \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " Nov 02 17:44:58.346858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.346833 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume\") pod \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\" (UID: \"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec\") " Nov 02 17:44:58.347144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.347114 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume" pod "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:58.347255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.347228 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume" pod "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:44:58.347691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.347668 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:58.347788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.347698 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:58.348993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.348962 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials" pod "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:58.349118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.349028 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2" pod "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" (UID: "dc162bf2-1da3-4a5e-880b-3f9230e5e2ec"). InnerVolumeSpecName "kube-api-access-99jt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:44:58.449446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.449411 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:58.449446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.449439 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jt2\" (UniqueName: \"kubernetes.io/projected/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec-kube-api-access-99jt2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:44:58.735194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.735159 5647 generic.go:356] "Generic (PLEG): container finished" podID="dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" containerID="88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9" exitCode=0 Nov 02 17:44:58.735406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.735204 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" Nov 02 17:44:58.735406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.735240 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" event={"ID":"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec","Type":"ContainerDied","Data":"88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9"} Nov 02 17:44:58.735406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.735293 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq" event={"ID":"dc162bf2-1da3-4a5e-880b-3f9230e5e2ec","Type":"ContainerDied","Data":"1afa25d50f881b9fd366264e7cbb52cce1a6135a8139e1cff012f78eeb37048d"} Nov 02 17:44:58.735406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.735316 5647 scope.go:122] "RemoveContainer" containerID="88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9" Nov 02 17:44:58.747250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.747229 5647 scope.go:122] "RemoveContainer" containerID="88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9" Nov 02 17:44:58.747538 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:44:58.747519 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9\": container with ID starting with 88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9 not found: ID does not exist" containerID="88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9" Nov 02 17:44:58.747580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.747549 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9"} err="failed to get container status \"88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9\": rpc error: code = NotFound desc = could not find container \"88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9\": container with ID starting with 88a0bff1ec51cd5ff1a6b624812ec8de1b84a4b86d1631fb36029fedbf58d3c9 not found: ID does not exist" Nov 02 17:44:58.905965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.905931 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq"] Nov 02 17:44:58.909325 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:58.909298 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7599585dbd-lsvnq"] Nov 02 17:44:59.091027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:59.090991 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" path="/var/lib/kubelet/pods/dc162bf2-1da3-4a5e-880b-3f9230e5e2ec/volumes" Nov 02 17:44:59.471920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:44:59.471829 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:45:00.170919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.170879 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4"] Nov 02 17:45:00.171725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.171709 5647 cpu_manager.go:414] "containerMap: removing container" podUID="dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" containerName="payload-processing" Nov 02 17:45:00.171810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.171765 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" containerName="payload-processing" Nov 02 17:45:00.171951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.171939 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc162bf2-1da3-4a5e-880b-3f9230e5e2ec" containerName="payload-processing" Nov 02 17:45:00.177202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.177173 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq"] Nov 02 17:45:00.177356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.177302 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:00.181429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.181407 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv"] Nov 02 17:45:00.181576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.181530 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:00.186164 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.185704 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:00.233403 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.233372 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:45:00.289561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.289525 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6"] Nov 02 17:45:00.367825 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.343612 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:00.368236 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.368187 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthqh\" (UniqueName: \"kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:00.399032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.398959 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:00.399367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.399343 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:00.405775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.399494 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5jj\" (UniqueName: \"kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj\") pod \"maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:00.413903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.413859 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tmw\" (UniqueName: \"kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:00.517825 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.496710 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:00.828946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.828851 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqjk\" (UniqueName: \"kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:00.906953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.906912 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:00.907161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.906962 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:00.907161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.907021 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:00.907161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.907057 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5jj\" (UniqueName: \"kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj\") pod \"maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:00.907161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.907072 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tmw\" (UniqueName: \"kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:00.907408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.907199 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:00.907408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.907238 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthqh\" (UniqueName: \"kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:00.911761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.908174 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:00.911761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.908592 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:00.911761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.909313 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:00.960830 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:00.960793 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv"] Nov 02 17:45:01.043797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.024670 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqjk\" (UniqueName: \"kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:01.043797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.024726 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:01.043797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.025852 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:01.077478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.077443 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthqh\" (UniqueName: \"kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:01.079273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.079177 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5jj\" (UniqueName: \"kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj\") pod \"maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:01.094788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.091129 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4"] Nov 02 17:45:01.094788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.091230 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv"] Nov 02 17:45:01.094788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.091913 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.094788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.094290 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:01.101160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.100848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tmw\" (UniqueName: \"kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:01.107886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.107857 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6"] Nov 02 17:45:01.114186 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.107989 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqjk\" (UniqueName: \"kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:01.119409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.119331 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv"] Nov 02 17:45:01.124009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.123764 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:01.133895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.127557 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-zwgsv\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.134091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.133993 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjjx9\" (UniqueName: \"kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9\") pod \"maas-api-key-cleanup-29894025-zwgsv\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.154963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.154934 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq"] Nov 02 17:45:01.162266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.162232 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:01.236842 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.235686 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-zwgsv\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.236842 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.235819 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjjx9\" (UniqueName: \"kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9\") pod \"maas-api-key-cleanup-29894025-zwgsv\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.237984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.237920 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894025-zwgsv\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.298051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.297983 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjjx9\" (UniqueName: \"kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9\") pod \"maas-api-key-cleanup-29894025-zwgsv\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.329804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.329691 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:01.475877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.473131 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:01.672337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.672292 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4"] Nov 02 17:45:01.693783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.692290 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq"] Nov 02 17:45:01.697330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.694967 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv"] Nov 02 17:45:01.799166 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:45:01.799122 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e834db_ed93_43ba_9788_ff671b2bb0e5.slice/crio-6b2ba842c4c36ea86511e6af5e0dfc2be1dd28b4b61ab24d336abbb66753f0a3 WatchSource:0}: Error finding container 6b2ba842c4c36ea86511e6af5e0dfc2be1dd28b4b61ab24d336abbb66753f0a3: Status 404 returned error can't find the container with id 6b2ba842c4c36ea86511e6af5e0dfc2be1dd28b4b61ab24d336abbb66753f0a3 Nov 02 17:45:01.863620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.863525 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6"] Nov 02 17:45:01.985485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.981502 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerStarted","Data":"fada48749333eaad29c6fc158594d58834647ab694fb6b7445c84560cec9e240"} Nov 02 17:45:01.985485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.982996 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerStarted","Data":"399dbca2d9d9ee2c597253c1847680b2a843288516f896d49270f88e2e181a6e"} Nov 02 17:45:01.985485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.984285 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerStarted","Data":"89d35d2b1b26c32b79e98c40edd79bdd0b73916da1f3e6139009a8834121b3e8"} Nov 02 17:45:01.985827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:01.985621 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerStarted","Data":"6b2ba842c4c36ea86511e6af5e0dfc2be1dd28b4b61ab24d336abbb66753f0a3"} Nov 02 17:45:02.163939 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:02.157119 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv"] Nov 02 17:45:03.339973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.335095 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:03.360062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.349820 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:03.360062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.357756 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerStarted","Data":"2823cc07da30d8fd76827edabe0f83471558169994625b2af47ace312a17d818"} Nov 02 17:45:03.360062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.357850 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerStarted","Data":"ac7aed745ae6b640187d5efe20dd5ddc5f8574a1d966409b29712f957c12c3e2"} Nov 02 17:45:03.383768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.363121 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerStarted","Data":"7614448dcd216ab479f7e74b1cba80d79d9d0da08acfa2f782a3c13ed61a8bcd"} Nov 02 17:45:03.603172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.554585 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:03.603172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.556047 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" podStartSLOduration=3.55602843 podStartE2EDuration="3.55602843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.528343283 +0000 UTC m=+1290.601870487" watchObservedRunningTime="2026-11-02 17:45:03.55602843 +0000 UTC m=+1290.629555634" Nov 02 17:45:03.623044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.604796 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerStarted","Data":"13c2e8cea1fde3f2ce51aa01002aef7bbdcf2aa6bb49ce045d81434f229fd007"} Nov 02 17:45:03.644806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.623541 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:03.677446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.677392 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:03.677835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.677754 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:03.698614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.698572 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerStarted","Data":"d18ae044105f0a83f91d13ec09bb2f9dd71993606ba1ca40afc8655f4f3d0de6"} Nov 02 17:45:03.754336 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.754271 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" podStartSLOduration=3.754248097 podStartE2EDuration="3.754248097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.719801037 +0000 UTC m=+1290.793328239" watchObservedRunningTime="2026-11-02 17:45:03.754248097 +0000 UTC m=+1290.827775304" Nov 02 17:45:03.922288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.922164 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerStarted","Data":"8eb497c571f3093f21caa8ccaff6147642fafe0e31a091182e39e62b51d0c52d"} Nov 02 17:45:03.923071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.923041 5647 scope.go:122] "RemoveContainer" containerID="8eb497c571f3093f21caa8ccaff6147642fafe0e31a091182e39e62b51d0c52d" Nov 02 17:45:03.971241 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.971198 5647 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:03.971651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:03.971618 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:04.350927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.350882 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:04.385787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.360705 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:45:04.386088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.382005 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Nov 02 17:45:04.387861 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.386719 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:45:04.437054 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.436986 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" podStartSLOduration=4.436964343 podStartE2EDuration="4.436964343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:03.904451916 +0000 UTC m=+1290.977979132" watchObservedRunningTime="2026-11-02 17:45:04.436964343 +0000 UTC m=+1291.510491547" Nov 02 17:45:04.440661 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.440585 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" podStartSLOduration=4.440572519 podStartE2EDuration="4.440572519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:04.029438645 +0000 UTC m=+1291.102965848" watchObservedRunningTime="2026-11-02 17:45:04.440572519 +0000 UTC m=+1291.514099727" Nov 02 17:45:04.544350 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:04.499113 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa11872_9c0b_4c70_924f_b5e7781ca49a.slice/crio-7614448dcd216ab479f7e74b1cba80d79d9d0da08acfa2f782a3c13ed61a8bcd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9b875d_fac9_4d5a_beb2_47f95956ba5c.slice/crio-d18ae044105f0a83f91d13ec09bb2f9dd71993606ba1ca40afc8655f4f3d0de6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e834db_ed93_43ba_9788_ff671b2bb0e5.slice/crio-8eb497c571f3093f21caa8ccaff6147642fafe0e31a091182e39e62b51d0c52d.scope\": RecentStats: unable to find data in memory cache]" Nov 02 17:45:04.563754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.556300 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:04.563754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.557148 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:45:04.626956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.626849 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:04.627659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.627634 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:45:05.005770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:04.997472 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:05.256155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.252641 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:45:05.268024 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.267985 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerID="d18ae044105f0a83f91d13ec09bb2f9dd71993606ba1ca40afc8655f4f3d0de6" exitCode=60 Nov 02 17:45:05.268194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.268091 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerDied","Data":"d18ae044105f0a83f91d13ec09bb2f9dd71993606ba1ca40afc8655f4f3d0de6"} Nov 02 17:45:05.268476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.268453 5647 scope.go:122] "RemoveContainer" containerID="d18ae044105f0a83f91d13ec09bb2f9dd71993606ba1ca40afc8655f4f3d0de6" Nov 02 17:45:05.274972 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.274945 5647 generic.go:356] "Generic (PLEG): container finished" podID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerID="8eb497c571f3093f21caa8ccaff6147642fafe0e31a091182e39e62b51d0c52d" exitCode=60 Nov 02 17:45:05.297789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.275162 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerDied","Data":"8eb497c571f3093f21caa8ccaff6147642fafe0e31a091182e39e62b51d0c52d"} Nov 02 17:45:05.297789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.293001 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 17:45:05.297789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.293050 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 17:45:05.297789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.293086 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 17:45:05.371580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.340293 5647 generic.go:356] "Generic (PLEG): container finished" podID="955a1ffa-c49a-424e-959d-07827f581c07" containerID="2823cc07da30d8fd76827edabe0f83471558169994625b2af47ace312a17d818" exitCode=60 Nov 02 17:45:05.371580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.340459 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerDied","Data":"2823cc07da30d8fd76827edabe0f83471558169994625b2af47ace312a17d818"} Nov 02 17:45:05.371580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.340950 5647 scope.go:122] "RemoveContainer" containerID="2823cc07da30d8fd76827edabe0f83471558169994625b2af47ace312a17d818" Nov 02 17:45:05.500824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.491025 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 17:45:05.602362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.589702 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:05.604131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.604089 5647 generic.go:356] "Generic (PLEG): container finished" podID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerID="7614448dcd216ab479f7e74b1cba80d79d9d0da08acfa2f782a3c13ed61a8bcd" exitCode=60 Nov 02 17:45:05.604606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.604571 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerDied","Data":"7614448dcd216ab479f7e74b1cba80d79d9d0da08acfa2f782a3c13ed61a8bcd"} Nov 02 17:45:05.605354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.605332 5647 scope.go:122] "RemoveContainer" containerID="7614448dcd216ab479f7e74b1cba80d79d9d0da08acfa2f782a3c13ed61a8bcd" Nov 02 17:45:05.696320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.696283 5647 generic.go:356] "Generic (PLEG): container finished" podID="60902803-b778-49d6-8f26-3d864711b284" containerID="13c2e8cea1fde3f2ce51aa01002aef7bbdcf2aa6bb49ce045d81434f229fd007" exitCode=60 Nov 02 17:45:05.716535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.704809 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerDied","Data":"13c2e8cea1fde3f2ce51aa01002aef7bbdcf2aa6bb49ce045d81434f229fd007"} Nov 02 17:45:05.716831 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.705399 5647 scope.go:122] "RemoveContainer" containerID="13c2e8cea1fde3f2ce51aa01002aef7bbdcf2aa6bb49ce045d81434f229fd007" Nov 02 17:45:05.786410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.786377 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-btngs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:45:05.786629 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.786605 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:45:05.878921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:05.878841 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 17:45:05.973774 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:05.973513 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:45:06.738585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.738540 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerID="d9e05bd1f08023c90180d5ba64adb8f0bdda7cd662efa6751d215db19f0f1504" exitCode=60 Nov 02 17:45:06.739071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.738622 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerDied","Data":"d9e05bd1f08023c90180d5ba64adb8f0bdda7cd662efa6751d215db19f0f1504"} Nov 02 17:45:06.739071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.738718 5647 scope.go:122] "RemoveContainer" containerID="d18ae044105f0a83f91d13ec09bb2f9dd71993606ba1ca40afc8655f4f3d0de6" Nov 02 17:45:06.739071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.739064 5647 scope.go:122] "RemoveContainer" containerID="d9e05bd1f08023c90180d5ba64adb8f0bdda7cd662efa6751d215db19f0f1504" Nov 02 17:45:06.739457 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:06.739430 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4_odh-ai-gateway-infra(1a9b875d-fac9-4d5a-beb2-47f95956ba5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" Nov 02 17:45:06.740845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.740822 5647 generic.go:356] "Generic (PLEG): container finished" podID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerID="4fb7f5b66d92f879cdeef6d2d980210de3248666be45a78053acd2568cad0934" exitCode=60 Nov 02 17:45:06.740964 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.740886 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerDied","Data":"4fb7f5b66d92f879cdeef6d2d980210de3248666be45a78053acd2568cad0934"} Nov 02 17:45:06.741241 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.741225 5647 scope.go:122] "RemoveContainer" containerID="4fb7f5b66d92f879cdeef6d2d980210de3248666be45a78053acd2568cad0934" Nov 02 17:45:06.741578 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:06.741559 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6_odh-ai-gateway-infra(09e834db-ed93-43ba-9788-ff671b2bb0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" Nov 02 17:45:06.743083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.743053 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerStarted","Data":"05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8"} Nov 02 17:45:06.745225 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.745202 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerStarted","Data":"3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5"} Nov 02 17:45:06.747238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.747218 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerStarted","Data":"3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54"} Nov 02 17:45:06.752641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:06.752608 5647 scope.go:122] "RemoveContainer" containerID="8eb497c571f3093f21caa8ccaff6147642fafe0e31a091182e39e62b51d0c52d" Nov 02 17:45:07.757224 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.757190 5647 scope.go:122] "RemoveContainer" containerID="4fb7f5b66d92f879cdeef6d2d980210de3248666be45a78053acd2568cad0934" Nov 02 17:45:07.757672 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:07.757541 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6_odh-ai-gateway-infra(09e834db-ed93-43ba-9788-ff671b2bb0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" Nov 02 17:45:07.759100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.759069 5647 generic.go:356] "Generic (PLEG): container finished" podID="955a1ffa-c49a-424e-959d-07827f581c07" containerID="05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8" exitCode=60 Nov 02 17:45:07.759221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.759192 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerDied","Data":"05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8"} Nov 02 17:45:07.759266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.759253 5647 scope.go:122] "RemoveContainer" containerID="2823cc07da30d8fd76827edabe0f83471558169994625b2af47ace312a17d818" Nov 02 17:45:07.759345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.759328 5647 scope.go:122] "RemoveContainer" containerID="05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8" Nov 02 17:45:07.759618 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:07.759581 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-zwgsv_odh-ai-gateway-infra(955a1ffa-c49a-424e-959d-07827f581c07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" podUID="955a1ffa-c49a-424e-959d-07827f581c07" Nov 02 17:45:07.763386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.763362 5647 generic.go:356] "Generic (PLEG): container finished" podID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerID="3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5" exitCode=60 Nov 02 17:45:07.763503 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.763483 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerDied","Data":"3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5"} Nov 02 17:45:07.763857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.763836 5647 scope.go:122] "RemoveContainer" containerID="3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5" Nov 02 17:45:07.764122 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:07.764103 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv_odh-ai-gateway-infra(efa11872-9c0b-4c70-924f-b5e7781ca49a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" Nov 02 17:45:07.766405 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.766377 5647 generic.go:356] "Generic (PLEG): container finished" podID="60902803-b778-49d6-8f26-3d864711b284" containerID="3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54" exitCode=60 Nov 02 17:45:07.766530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.766509 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerDied","Data":"3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54"} Nov 02 17:45:07.766770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.766721 5647 scope.go:122] "RemoveContainer" containerID="3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54" Nov 02 17:45:07.767037 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:07.767014 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq_odh-ai-gateway-infra(60902803-b778-49d6-8f26-3d864711b284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" podUID="60902803-b778-49d6-8f26-3d864711b284" Nov 02 17:45:07.768619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.768604 5647 scope.go:122] "RemoveContainer" containerID="d9e05bd1f08023c90180d5ba64adb8f0bdda7cd662efa6751d215db19f0f1504" Nov 02 17:45:07.768842 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:07.768825 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4_odh-ai-gateway-infra(1a9b875d-fac9-4d5a-beb2-47f95956ba5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" Nov 02 17:45:07.773882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.773859 5647 scope.go:122] "RemoveContainer" containerID="7614448dcd216ab479f7e74b1cba80d79d9d0da08acfa2f782a3c13ed61a8bcd" Nov 02 17:45:07.788237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:07.788216 5647 scope.go:122] "RemoveContainer" containerID="13c2e8cea1fde3f2ce51aa01002aef7bbdcf2aa6bb49ce045d81434f229fd007" Nov 02 17:45:08.777946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:08.777915 5647 scope.go:122] "RemoveContainer" containerID="3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5" Nov 02 17:45:08.778419 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:08.778287 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv_odh-ai-gateway-infra(efa11872-9c0b-4c70-924f-b5e7781ca49a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" Nov 02 17:45:08.779708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:08.779690 5647 scope.go:122] "RemoveContainer" containerID="3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54" Nov 02 17:45:08.780136 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:08.780116 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq_odh-ai-gateway-infra(60902803-b778-49d6-8f26-3d864711b284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" podUID="60902803-b778-49d6-8f26-3d864711b284" Nov 02 17:45:08.781543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:08.781526 5647 scope.go:122] "RemoveContainer" containerID="05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8" Nov 02 17:45:08.781790 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:08.781773 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-zwgsv_odh-ai-gateway-infra(955a1ffa-c49a-424e-959d-07827f581c07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" podUID="955a1ffa-c49a-424e-959d-07827f581c07" Nov 02 17:45:11.695851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.694236 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": read tcp 10.134.0.2:48790->10.134.0.98:15021: read: connection reset by peer" start-of-body= Nov 02 17:45:11.695851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.694300 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": read tcp 10.134.0.2:48790->10.134.0.98:15021: read: connection reset by peer" Nov 02 17:45:11.806332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.806299 5647 generic.go:356] "Generic (PLEG): container finished" podID="576113f5-04da-4871-980d-c101f55b1622" containerID="fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c" exitCode=0 Nov 02 17:45:11.806524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.806373 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" event={"ID":"576113f5-04da-4871-980d-c101f55b1622","Type":"ContainerDied","Data":"fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c"} Nov 02 17:45:11.806853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.806817 5647 scope.go:122] "RemoveContainer" containerID="fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c" Nov 02 17:45:11.808293 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.808263 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:45:11.808384 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.808332 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:45:11.808384 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:11.808367 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:45:12.833163 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:12.833126 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" event={"ID":"576113f5-04da-4871-980d-c101f55b1622","Type":"ContainerStarted","Data":"45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b"} Nov 02 17:45:13.374603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:13.374567 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:45:13.374814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:13.374792 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:45:13.374864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:13.374834 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 17:45:14.374140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:14.374097 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:45:14.374553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:14.374154 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 17:45:15.374642 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:15.374603 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:45:15.375129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:15.374676 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 17:45:16.374485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:16.374442 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:45:16.374680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:16.374494 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 17:45:17.373913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:17.373881 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 02 17:45:17.374169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:17.374148 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 02 17:45:18.074677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.074643 5647 scope.go:122] "RemoveContainer" containerID="4fb7f5b66d92f879cdeef6d2d980210de3248666be45a78053acd2568cad0934" Nov 02 17:45:18.075149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.075065 5647 scope.go:122] "RemoveContainer" containerID="3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5" Nov 02 17:45:18.378912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.378872 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:45:18.379358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.379346 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:45:18.380137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.380115 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:45:18.880017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.879983 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerStarted","Data":"ea0f7821cd850ca9dc158482bf6128886dec74a408f54b21c7189764617c0cc3"} Nov 02 17:45:18.882330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.882300 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerStarted","Data":"49d5317a87a32beb162363e5c9eec0c3cf97035f4fbd62995cda3b3d79e4db32"} Nov 02 17:45:18.998958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:18.998905 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" podStartSLOduration=18.998889973 podStartE2EDuration="18.998889973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:18.994875492 +0000 UTC m=+1306.068402695" watchObservedRunningTime="2026-11-02 17:45:18.998889973 +0000 UTC m=+1306.072417216" Nov 02 17:45:19.074747 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.074700 5647 scope.go:122] "RemoveContainer" containerID="d9e05bd1f08023c90180d5ba64adb8f0bdda7cd662efa6751d215db19f0f1504" Nov 02 17:45:19.075219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.074987 5647 scope.go:122] "RemoveContainer" containerID="05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8" Nov 02 17:45:19.891613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.891580 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerStarted","Data":"0529201b221403254900933266c7efceaeb8ba971b2c1f03479ebd6a9151964c"} Nov 02 17:45:19.893698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.893675 5647 generic.go:356] "Generic (PLEG): container finished" podID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerID="ea0f7821cd850ca9dc158482bf6128886dec74a408f54b21c7189764617c0cc3" exitCode=60 Nov 02 17:45:19.893869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.893758 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerDied","Data":"ea0f7821cd850ca9dc158482bf6128886dec74a408f54b21c7189764617c0cc3"} Nov 02 17:45:19.893869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.893818 5647 scope.go:122] "RemoveContainer" containerID="4fb7f5b66d92f879cdeef6d2d980210de3248666be45a78053acd2568cad0934" Nov 02 17:45:19.894156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.894139 5647 scope.go:122] "RemoveContainer" containerID="ea0f7821cd850ca9dc158482bf6128886dec74a408f54b21c7189764617c0cc3" Nov 02 17:45:19.894465 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:19.894440 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6_odh-ai-gateway-infra(09e834db-ed93-43ba-9788-ff671b2bb0e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" Nov 02 17:45:19.896147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.896120 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerStarted","Data":"a6f5f33b8dd00bb3560248f3fd0d9d91547fa253a9ca99d63202bdad584c0849"} Nov 02 17:45:19.898452 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.898424 5647 generic.go:356] "Generic (PLEG): container finished" podID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerID="49d5317a87a32beb162363e5c9eec0c3cf97035f4fbd62995cda3b3d79e4db32" exitCode=60 Nov 02 17:45:19.898558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.898451 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerDied","Data":"49d5317a87a32beb162363e5c9eec0c3cf97035f4fbd62995cda3b3d79e4db32"} Nov 02 17:45:19.898798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.898783 5647 scope.go:122] "RemoveContainer" containerID="49d5317a87a32beb162363e5c9eec0c3cf97035f4fbd62995cda3b3d79e4db32" Nov 02 17:45:19.899037 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:19.899020 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv_odh-ai-gateway-infra(efa11872-9c0b-4c70-924f-b5e7781ca49a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" Nov 02 17:45:19.908828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:19.908793 5647 scope.go:122] "RemoveContainer" containerID="3b8af00ef32f10388c738ede401e9f7654b3b51790856616186d757e948d07d5" Nov 02 17:45:20.038789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.033824 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6"] Nov 02 17:45:20.081491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.079946 5647 scope.go:122] "RemoveContainer" containerID="3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54" Nov 02 17:45:20.122859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.122828 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv"] Nov 02 17:45:20.909199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.909158 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerID="0529201b221403254900933266c7efceaeb8ba971b2c1f03479ebd6a9151964c" exitCode=60 Nov 02 17:45:20.909404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.909231 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerDied","Data":"0529201b221403254900933266c7efceaeb8ba971b2c1f03479ebd6a9151964c"} Nov 02 17:45:20.909404 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.909318 5647 scope.go:122] "RemoveContainer" containerID="d9e05bd1f08023c90180d5ba64adb8f0bdda7cd662efa6751d215db19f0f1504" Nov 02 17:45:20.909794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.909777 5647 scope.go:122] "RemoveContainer" containerID="0529201b221403254900933266c7efceaeb8ba971b2c1f03479ebd6a9151964c" Nov 02 17:45:20.910174 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:20.910138 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4_odh-ai-gateway-infra(1a9b875d-fac9-4d5a-beb2-47f95956ba5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" Nov 02 17:45:20.913250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.913227 5647 generic.go:356] "Generic (PLEG): container finished" podID="955a1ffa-c49a-424e-959d-07827f581c07" containerID="a6f5f33b8dd00bb3560248f3fd0d9d91547fa253a9ca99d63202bdad584c0849" exitCode=60 Nov 02 17:45:20.913358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.913299 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerDied","Data":"a6f5f33b8dd00bb3560248f3fd0d9d91547fa253a9ca99d63202bdad584c0849"} Nov 02 17:45:20.913665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.913646 5647 scope.go:122] "RemoveContainer" containerID="a6f5f33b8dd00bb3560248f3fd0d9d91547fa253a9ca99d63202bdad584c0849" Nov 02 17:45:20.913922 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:20.913900 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894025-zwgsv_odh-ai-gateway-infra(955a1ffa-c49a-424e-959d-07827f581c07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" podUID="955a1ffa-c49a-424e-959d-07827f581c07" Nov 02 17:45:20.918046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.918021 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerStarted","Data":"09a661bb053ae8a36c9ac6072cda76fdee89e8d3dd9fd0f82f02144f12420f69"} Nov 02 17:45:20.923857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:20.923834 5647 scope.go:122] "RemoveContainer" containerID="05c027e1c39d7595d9901bef78ca1d09acfd56b262868b4024dce5f0d51346c8" Nov 02 17:45:21.024396 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.024362 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4"] Nov 02 17:45:21.103144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.103112 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:21.165858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.165794 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:21.217102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.217066 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca\") pod \"efa11872-9c0b-4c70-924f-b5e7781ca49a\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " Nov 02 17:45:21.217306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.217148 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca\") pod \"09e834db-ed93-43ba-9788-ff671b2bb0e5\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " Nov 02 17:45:21.217306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.217212 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk\" (UniqueName: \"kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk\") pod \"09e834db-ed93-43ba-9788-ff671b2bb0e5\" (UID: \"09e834db-ed93-43ba-9788-ff671b2bb0e5\") " Nov 02 17:45:21.217431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.217361 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw\" (UniqueName: \"kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw\") pod \"efa11872-9c0b-4c70-924f-b5e7781ca49a\" (UID: \"efa11872-9c0b-4c70-924f-b5e7781ca49a\") " Nov 02 17:45:21.217489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.217462 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca" pod "efa11872-9c0b-4c70-924f-b5e7781ca49a" (UID: "efa11872-9c0b-4c70-924f-b5e7781ca49a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:21.217581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.217560 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca" pod "09e834db-ed93-43ba-9788-ff671b2bb0e5" (UID: "09e834db-ed93-43ba-9788-ff671b2bb0e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:21.218085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.218062 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/efa11872-9c0b-4c70-924f-b5e7781ca49a-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:21.218085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.218085 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e834db-ed93-43ba-9788-ff671b2bb0e5-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:21.219550 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.219528 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw" pod "efa11872-9c0b-4c70-924f-b5e7781ca49a" (UID: "efa11872-9c0b-4c70-924f-b5e7781ca49a"). InnerVolumeSpecName "kube-api-access-n6tmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:21.219621 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.219552 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk" pod "09e834db-ed93-43ba-9788-ff671b2bb0e5" (UID: "09e834db-ed93-43ba-9788-ff671b2bb0e5"). InnerVolumeSpecName "kube-api-access-hxqjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:21.319539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.319502 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqjk\" (UniqueName: \"kubernetes.io/projected/09e834db-ed93-43ba-9788-ff671b2bb0e5-kube-api-access-hxqjk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:21.319539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.319533 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tmw\" (UniqueName: \"kubernetes.io/projected/efa11872-9c0b-4c70-924f-b5e7781ca49a-kube-api-access-n6tmw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:21.393178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.393143 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv"] Nov 02 17:45:21.927149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.927118 5647 generic.go:356] "Generic (PLEG): container finished" podID="60902803-b778-49d6-8f26-3d864711b284" containerID="09a661bb053ae8a36c9ac6072cda76fdee89e8d3dd9fd0f82f02144f12420f69" exitCode=60 Nov 02 17:45:21.927370 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.927202 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerDied","Data":"09a661bb053ae8a36c9ac6072cda76fdee89e8d3dd9fd0f82f02144f12420f69"} Nov 02 17:45:21.927370 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.927269 5647 scope.go:122] "RemoveContainer" containerID="3bcfbf24f72c98a29700a6caddd2b85cf4ddf9b324a54896cd583a8a3c272d54" Nov 02 17:45:21.927665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.927645 5647 scope.go:122] "RemoveContainer" containerID="09a661bb053ae8a36c9ac6072cda76fdee89e8d3dd9fd0f82f02144f12420f69" Nov 02 17:45:21.928024 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:21.928004 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq_odh-ai-gateway-infra(60902803-b778-49d6-8f26-3d864711b284)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" podUID="60902803-b778-49d6-8f26-3d864711b284" Nov 02 17:45:21.930608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.930587 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" Nov 02 17:45:21.930608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.930601 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6" event={"ID":"09e834db-ed93-43ba-9788-ff671b2bb0e5","Type":"ContainerDied","Data":"6b2ba842c4c36ea86511e6af5e0dfc2be1dd28b4b61ab24d336abbb66753f0a3"} Nov 02 17:45:21.934075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.934047 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" event={"ID":"efa11872-9c0b-4c70-924f-b5e7781ca49a","Type":"ContainerDied","Data":"fada48749333eaad29c6fc158594d58834647ab694fb6b7445c84560cec9e240"} Nov 02 17:45:21.934169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.934085 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv" Nov 02 17:45:21.943106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.943080 5647 scope.go:122] "RemoveContainer" containerID="ea0f7821cd850ca9dc158482bf6128886dec74a408f54b21c7189764617c0cc3" Nov 02 17:45:21.956248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:21.956222 5647 scope.go:122] "RemoveContainer" containerID="49d5317a87a32beb162363e5c9eec0c3cf97035f4fbd62995cda3b3d79e4db32" Nov 02 17:45:22.129802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.129470 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:22.131678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.131653 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:22.231144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.231051 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca\") pod \"955a1ffa-c49a-424e-959d-07827f581c07\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " Nov 02 17:45:22.231300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.231169 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9\" (UniqueName: \"kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9\") pod \"955a1ffa-c49a-424e-959d-07827f581c07\" (UID: \"955a1ffa-c49a-424e-959d-07827f581c07\") " Nov 02 17:45:22.231356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.231303 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca\") pod \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " Nov 02 17:45:22.231447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.231420 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca" pod "955a1ffa-c49a-424e-959d-07827f581c07" (UID: "955a1ffa-c49a-424e-959d-07827f581c07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:22.231485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.231451 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh\" (UniqueName: \"kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh\") pod \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\" (UID: \"1a9b875d-fac9-4d5a-beb2-47f95956ba5c\") " Nov 02 17:45:22.231778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.231724 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca" pod "1a9b875d-fac9-4d5a-beb2-47f95956ba5c" (UID: "1a9b875d-fac9-4d5a-beb2-47f95956ba5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:22.232069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.232051 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.232128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.232076 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/955a1ffa-c49a-424e-959d-07827f581c07-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.233610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.233585 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh" pod "1a9b875d-fac9-4d5a-beb2-47f95956ba5c" (UID: "1a9b875d-fac9-4d5a-beb2-47f95956ba5c"). InnerVolumeSpecName "kube-api-access-sthqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:22.233720 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.233700 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9" pod "955a1ffa-c49a-424e-959d-07827f581c07" (UID: "955a1ffa-c49a-424e-959d-07827f581c07"). InnerVolumeSpecName "kube-api-access-wjjx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:22.273958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.273922 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6"] Nov 02 17:45:22.277546 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.277519 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ad655-29894025-f7mn6"] Nov 02 17:45:22.332897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.332859 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthqh\" (UniqueName: \"kubernetes.io/projected/1a9b875d-fac9-4d5a-beb2-47f95956ba5c-kube-api-access-sthqh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.332897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.332888 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjjx9\" (UniqueName: \"kubernetes.io/projected/955a1ffa-c49a-424e-959d-07827f581c07-kube-api-access-wjjx9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:22.390330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.390294 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq"] Nov 02 17:45:22.504155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.504082 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv"] Nov 02 17:45:22.507601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.507577 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894025-wkwhv"] Nov 02 17:45:22.943031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.942991 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" Nov 02 17:45:22.943221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.942997 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4" event={"ID":"1a9b875d-fac9-4d5a-beb2-47f95956ba5c","Type":"ContainerDied","Data":"89d35d2b1b26c32b79e98c40edd79bdd0b73916da1f3e6139009a8834121b3e8"} Nov 02 17:45:22.943221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.943128 5647 scope.go:122] "RemoveContainer" containerID="0529201b221403254900933266c7efceaeb8ba971b2c1f03479ebd6a9151964c" Nov 02 17:45:22.947270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.947247 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" Nov 02 17:45:22.947270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.947269 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv" event={"ID":"955a1ffa-c49a-424e-959d-07827f581c07","Type":"ContainerDied","Data":"ac7aed745ae6b640187d5efe20dd5ddc5f8574a1d966409b29712f957c12c3e2"} Nov 02 17:45:22.964597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:22.964577 5647 scope.go:122] "RemoveContainer" containerID="a6f5f33b8dd00bb3560248f3fd0d9d91547fa253a9ca99d63202bdad584c0849" Nov 02 17:45:23.083259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.083227 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" path="/var/lib/kubelet/pods/09e834db-ed93-43ba-9788-ff671b2bb0e5/volumes" Nov 02 17:45:23.083703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.083689 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" path="/var/lib/kubelet/pods/efa11872-9c0b-4c70-924f-b5e7781ca49a/volumes" Nov 02 17:45:23.112832 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.112800 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4"] Nov 02 17:45:23.118706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.118682 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894025-bxlv4"] Nov 02 17:45:23.125898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.125872 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:23.245114 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.245013 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj\" (UniqueName: \"kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj\") pod \"60902803-b778-49d6-8f26-3d864711b284\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " Nov 02 17:45:23.245571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.245197 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca\") pod \"60902803-b778-49d6-8f26-3d864711b284\" (UID: \"60902803-b778-49d6-8f26-3d864711b284\") " Nov 02 17:45:23.247997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.247958 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca" pod "60902803-b778-49d6-8f26-3d864711b284" (UID: "60902803-b778-49d6-8f26-3d864711b284"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:45:23.256614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.248953 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60902803-b778-49d6-8f26-3d864711b284-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:23.256779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.256689 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj" pod "60902803-b778-49d6-8f26-3d864711b284" (UID: "60902803-b778-49d6-8f26-3d864711b284"). InnerVolumeSpecName "kube-api-access-9z5jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:45:23.344330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.344291 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv"] Nov 02 17:45:23.347788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.347759 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894025-zwgsv"] Nov 02 17:45:23.350430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.350398 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5jj\" (UniqueName: \"kubernetes.io/projected/60902803-b778-49d6-8f26-3d864711b284-kube-api-access-9z5jj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:45:23.959306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.959276 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" Nov 02 17:45:23.959306 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.959289 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq" event={"ID":"60902803-b778-49d6-8f26-3d864711b284","Type":"ContainerDied","Data":"399dbca2d9d9ee2c597253c1847680b2a843288516f896d49270f88e2e181a6e"} Nov 02 17:45:23.959560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:23.959367 5647 scope.go:122] "RemoveContainer" containerID="09a661bb053ae8a36c9ac6072cda76fdee89e8d3dd9fd0f82f02144f12420f69" Nov 02 17:45:24.173591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:24.173545 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq"] Nov 02 17:45:24.189759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:24.185336 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8cbedb9f-29894025-g29nq"] Nov 02 17:45:25.081875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:25.081837 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" path="/var/lib/kubelet/pods/1a9b875d-fac9-4d5a-beb2-47f95956ba5c/volumes" Nov 02 17:45:25.082302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:25.082253 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60902803-b778-49d6-8f26-3d864711b284" path="/var/lib/kubelet/pods/60902803-b778-49d6-8f26-3d864711b284/volumes" Nov 02 17:45:25.082644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:25.082630 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955a1ffa-c49a-424e-959d-07827f581c07" path="/var/lib/kubelet/pods/955a1ffa-c49a-424e-959d-07827f581c07/volumes" Nov 02 17:45:32.993472 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:32.993437 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:45:34.502760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:34.502633 5647 scope.go:122] "RemoveContainer" containerID="e62bb2cefb5bb60b74ca8189581622fa0e9725d96ed21716e85c12cb3881ef45" Nov 02 17:45:34.523295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:34.523265 5647 scope.go:122] "RemoveContainer" containerID="836c82859462236f2eef4faeef705e99c317d61374a69780dd5b64d8cd0c59a6" Nov 02 17:45:34.541141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:45:34.540675 5647 scope.go:122] "RemoveContainer" containerID="c36a0b9988e475997842d3db060c75987bc1699ecaf7de062f10204efe948050" Nov 02 17:45:36.065764 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:45:36.063160 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:46:00.476246 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476209 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl"] Nov 02 17:46:00.476789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476771 5647 cpu_manager.go:414] "containerMap: removing container" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:00.476837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476792 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:00.476837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476810 5647 cpu_manager.go:414] "containerMap: removing container" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.476837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476817 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476843 5647 cpu_manager.go:414] "containerMap: removing container" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476848 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476865 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476874 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476883 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476888 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476895 5647 cpu_manager.go:414] "containerMap: removing container" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476900 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476906 5647 cpu_manager.go:414] "containerMap: removing container" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476911 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476917 5647 cpu_manager.go:414] "containerMap: removing container" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476921 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476930 5647 cpu_manager.go:414] "containerMap: removing container" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:00.476929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476935 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476946 5647 cpu_manager.go:414] "containerMap: removing container" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476951 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476960 5647 cpu_manager.go:414] "containerMap: removing container" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.476965 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477068 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477075 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477081 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477087 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477092 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477097 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477102 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477106 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477112 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477118 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477123 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa11872-9c0b-4c70-924f-b5e7781ca49a" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477128 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:00.477339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.477133 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:00.480850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.480832 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.538717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.538684 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl"] Nov 02 17:46:00.579530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.579498 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x58wj\" (UniqueName: \"kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.579692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.579633 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.579767 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.579701 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.583243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.583220 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:46:00.583550 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.583530 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:46:00.680605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.680568 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.680846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.680643 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.680846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.680807 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x58wj\" (UniqueName: \"kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.681415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.681389 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.683196 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.683173 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.739004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.738938 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x58wj\" (UniqueName: \"kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj\") pod \"maas-api-revoke-sub-6159bd21-kl7rl\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:00.794330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:00.794301 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:01.173441 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:46:01.173407 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e4f831_1522_4092_bcf9_514e9a243b16.slice/crio-5f290e8ad9218aa9aa0ac7d72ac57bbb951ad2cf9322862e745141dc79d16283 WatchSource:0}: Error finding container 5f290e8ad9218aa9aa0ac7d72ac57bbb951ad2cf9322862e745141dc79d16283: Status 404 returned error can't find the container with id 5f290e8ad9218aa9aa0ac7d72ac57bbb951ad2cf9322862e745141dc79d16283 Nov 02 17:46:01.173612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:01.173583 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl"] Nov 02 17:46:01.550992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:01.550946 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" event={"ID":"49e4f831-1522-4092-bcf9-514e9a243b16","Type":"ContainerStarted","Data":"07fb9680b27b8abc422938cfac6cb55f9886fb1e2582b746d9973dff29dc1abd"} Nov 02 17:46:01.551463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:01.551003 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" event={"ID":"49e4f831-1522-4092-bcf9-514e9a243b16","Type":"ContainerStarted","Data":"5f290e8ad9218aa9aa0ac7d72ac57bbb951ad2cf9322862e745141dc79d16283"} Nov 02 17:46:01.666930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:01.666877 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" podStartSLOduration=1.666861184 podStartE2EDuration="1.666861184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:01.665467131 +0000 UTC m=+1348.738994335" watchObservedRunningTime="2026-11-02 17:46:01.666861184 +0000 UTC m=+1348.740388387" Nov 02 17:46:02.559911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:02.559875 5647 generic.go:356] "Generic (PLEG): container finished" podID="49e4f831-1522-4092-bcf9-514e9a243b16" containerID="07fb9680b27b8abc422938cfac6cb55f9886fb1e2582b746d9973dff29dc1abd" exitCode=0 Nov 02 17:46:02.560346 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:02.559957 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" event={"ID":"49e4f831-1522-4092-bcf9-514e9a243b16","Type":"ContainerDied","Data":"07fb9680b27b8abc422938cfac6cb55f9886fb1e2582b746d9973dff29dc1abd"} Nov 02 17:46:03.737512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.737488 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:03.845500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.845475 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj\" (UniqueName: \"kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj\") pod \"49e4f831-1522-4092-bcf9-514e9a243b16\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " Nov 02 17:46:03.845682 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.845513 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca\") pod \"49e4f831-1522-4092-bcf9-514e9a243b16\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " Nov 02 17:46:03.845778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.845678 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca\") pod \"49e4f831-1522-4092-bcf9-514e9a243b16\" (UID: \"49e4f831-1522-4092-bcf9-514e9a243b16\") " Nov 02 17:46:03.846051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.846024 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca" pod "49e4f831-1522-4092-bcf9-514e9a243b16" (UID: "49e4f831-1522-4092-bcf9-514e9a243b16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:46:03.846315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.846292 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:46:03.847793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.847765 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca" pod "49e4f831-1522-4092-bcf9-514e9a243b16" (UID: "49e4f831-1522-4092-bcf9-514e9a243b16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:46:03.847878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.847824 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj" pod "49e4f831-1522-4092-bcf9-514e9a243b16" (UID: "49e4f831-1522-4092-bcf9-514e9a243b16"). InnerVolumeSpecName "kube-api-access-x58wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:46:03.951574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.951540 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x58wj\" (UniqueName: \"kubernetes.io/projected/49e4f831-1522-4092-bcf9-514e9a243b16-kube-api-access-x58wj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:46:03.951822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:03.951805 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49e4f831-1522-4092-bcf9-514e9a243b16-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:46:04.576498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:04.576471 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" Nov 02 17:46:04.576692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:04.576510 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl" event={"ID":"49e4f831-1522-4092-bcf9-514e9a243b16","Type":"ContainerDied","Data":"5f290e8ad9218aa9aa0ac7d72ac57bbb951ad2cf9322862e745141dc79d16283"} Nov 02 17:46:04.576692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:04.576565 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f290e8ad9218aa9aa0ac7d72ac57bbb951ad2cf9322862e745141dc79d16283" Nov 02 17:46:06.026338 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:46:06.026297 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:46:17.721332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:17.721282 5647 generic.go:356] "Generic (PLEG): container finished" podID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerID="02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826" exitCode=0 Nov 02 17:46:17.721959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:17.721455 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" event={"ID":"430deb5a-0705-48e0-a500-2fa09af8a42d","Type":"ContainerDied","Data":"02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826"} Nov 02 17:46:17.743756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:17.727605 5647 scope.go:122] "RemoveContainer" containerID="02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826" Nov 02 17:46:17.743756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:17.738697 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:17.743756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:17.738834 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:17.743756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:17.738887 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:18.737126 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:18.737085 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" event={"ID":"430deb5a-0705-48e0-a500-2fa09af8a42d","Type":"ContainerStarted","Data":"77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143"} Nov 02 17:46:19.414757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:19.414701 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:46:19.419763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:19.417844 5647 patch_prober.go:29] interesting pod/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:46:19.419763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:19.417900 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:46:20.427174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:20.427137 5647 patch_prober.go:29] interesting pod/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:46:20.427598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:20.427191 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:46:21.422905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:21.422852 5647 patch_prober.go:29] interesting pod/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:46:21.423196 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:21.423167 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:46:22.426910 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:22.426866 5647 patch_prober.go:29] interesting pod/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:46:22.427447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:22.426934 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:46:23.417282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:23.417245 5647 patch_prober.go:29] interesting pod/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Nov 02 17:46:23.417596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:23.417561 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Nov 02 17:46:24.410341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.410311 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:46:24.410796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.410776 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:46:24.411545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.411525 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:46:24.860668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.860633 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj"] Nov 02 17:46:24.861239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861221 5647 cpu_manager.go:414] "containerMap: removing container" podUID="49e4f831-1522-4092-bcf9-514e9a243b16" containerName="revoke-keys" Nov 02 17:46:24.861295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861240 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e4f831-1522-4092-bcf9-514e9a243b16" containerName="revoke-keys" Nov 02 17:46:24.861295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861261 5647 cpu_manager.go:414] "containerMap: removing container" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:24.861295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861266 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:24.861295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861287 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:24.861295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861292 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9b875d-fac9-4d5a-beb2-47f95956ba5c" containerName="cleanup" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861305 5647 cpu_manager.go:414] "containerMap: removing container" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861310 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861331 5647 cpu_manager.go:414] "containerMap: removing container" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861336 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e834db-ed93-43ba-9788-ff671b2bb0e5" containerName="cleanup" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861423 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e4f831-1522-4092-bcf9-514e9a243b16" containerName="revoke-keys" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861436 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="60902803-b778-49d6-8f26-3d864711b284" containerName="cleanup" Nov 02 17:46:24.861454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.861444 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="955a1ffa-c49a-424e-959d-07827f581c07" containerName="cleanup" Nov 02 17:46:24.863988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.863970 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:24.955196 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:24.955165 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-494cff-openshift-default-dockercfg-ghppm\"" Nov 02 17:46:25.157079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.156961 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.157525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.157485 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.157711 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.157696 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.157896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.157880 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.158177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.158160 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.158317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.158302 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.158443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.158429 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.158666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.158647 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdnt\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.158866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.158836 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.252923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.226796 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj"] Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260346 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260400 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260432 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260463 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260594 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260620 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.260754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260660 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.261407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260811 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdnt\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.261407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.260873 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.264985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.263177 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.273770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.268386 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.273770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.268643 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.273770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.269124 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.273770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.269900 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.273770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.270438 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.312759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.306269 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.364604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.359378 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.364604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.363299 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdnt\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt\") pod \"e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.477689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.477601 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:25.908113 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.908076 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj"] Nov 02 17:46:25.908369 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:46:25.908345 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136f7b71_9fe6_41cf_9d10_bbd60ff34d1a.slice/crio-672d28f4f0c297791e2f71c2d3e46ad2592da29217937beba32bdc8b6a4e18ee WatchSource:0}: Error finding container 672d28f4f0c297791e2f71c2d3e46ad2592da29217937beba32bdc8b6a4e18ee: Status 404 returned error can't find the container with id 672d28f4f0c297791e2f71c2d3e46ad2592da29217937beba32bdc8b6a4e18ee Nov 02 17:46:25.910797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.910763 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:25.910886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.910841 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:25.910962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:25.910912 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:26.553606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.553567 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx"] Nov 02 17:46:26.556667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.556646 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.617756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.610645 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-215cbd-openshift-default-dockercfg-jnpgh\"" Nov 02 17:46:26.850009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.849968 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx"] Nov 02 17:46:26.865035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.864978 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.867494 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.867450 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.868045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.868018 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.875472 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.875439 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.876109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.876082 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.877147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.877120 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.877333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.877315 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.877497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.877480 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.877638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.877623 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z797n\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983349 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983415 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983491 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983546 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z797n\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983673 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983707 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983782 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983822 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.995824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.983916 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.996442 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.996305 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.996711 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.996669 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.997309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.997279 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:26.997579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:26.997557 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.029684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.029646 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.055759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.053778 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.075948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.075913 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.111079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.110989 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.117714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.117664 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z797n\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n\") pod \"e2e-api-a-215cbd-openshift-default-7449444b59-77cnx\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.123774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.122989 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" event={"ID":"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a","Type":"ContainerStarted","Data":"034d94b5082456eb0b87b52ade49347a98c6a6b48b3fb515eaaf9f6557e23a4a"} Nov 02 17:46:27.123774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.123056 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" event={"ID":"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a","Type":"ContainerStarted","Data":"672d28f4f0c297791e2f71c2d3e46ad2592da29217937beba32bdc8b6a4e18ee"} Nov 02 17:46:27.170881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.170838 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:27.477991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.477855 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:27.483802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.483776 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:27.606974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.606926 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podStartSLOduration=3.606911701 podStartE2EDuration="3.606911701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:27.239327116 +0000 UTC m=+1374.312854356" watchObservedRunningTime="2026-11-02 17:46:27.606911701 +0000 UTC m=+1374.680438904" Nov 02 17:46:27.607421 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:46:27.607389 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a491df_b1ac_4bd6_9a99_e114a01c35dc.slice/crio-96265f067301b5829932392080e584479477d9cfa7be57f0468192419d94cdca WatchSource:0}: Error finding container 96265f067301b5829932392080e584479477d9cfa7be57f0468192419d94cdca: Status 404 returned error can't find the container with id 96265f067301b5829932392080e584479477d9cfa7be57f0468192419d94cdca Nov 02 17:46:27.607529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.607441 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx"] Nov 02 17:46:27.610645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.610612 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:27.610771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.610677 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:27.610771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:27.610706 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:46:28.123372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.123329 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" event={"ID":"e6a491df-b1ac-4bd6-9a99-e114a01c35dc","Type":"ContainerStarted","Data":"201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2"} Nov 02 17:46:28.123562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.123394 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" event={"ID":"e6a491df-b1ac-4bd6-9a99-e114a01c35dc","Type":"ContainerStarted","Data":"96265f067301b5829932392080e584479477d9cfa7be57f0468192419d94cdca"} Nov 02 17:46:28.123562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.123501 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:28.124592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.124569 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:46:28.171257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.171228 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:28.176958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.176929 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:28.244092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:28.244038 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podStartSLOduration=2.244025087 podStartE2EDuration="2.244025087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:28.242162811 +0000 UTC m=+1375.315690016" watchObservedRunningTime="2026-11-02 17:46:28.244025087 +0000 UTC m=+1375.317552290" Nov 02 17:46:29.130659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:29.130629 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:29.131814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:29.131793 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:46:35.947721 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:46:35.947672 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:46:38.792535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:38.792503 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:46:42.965242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:42.965193 5647 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:46:43.287370 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:46:43.287268 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:47:05.947677 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:05.947638 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:47:16.049678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:16.049642 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt"] Nov 02 17:47:16.100492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:16.100463 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe8f212b-782bt"] Nov 02 17:47:17.081299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:17.081265 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb72637d-f6a7-4c8b-8adc-2c2952835878" path="/var/lib/kubelet/pods/fb72637d-f6a7-4c8b-8adc-2c2952835878/volumes" Nov 02 17:47:24.772880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:24.772844 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:47:27.395589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.395553 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5"] Nov 02 17:47:27.399983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.399964 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.456975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.456941 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5"] Nov 02 17:47:27.502651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.502620 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:47:27.502651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.502630 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:47:27.552830 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.552723 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.553008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.552964 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9tx\" (UniqueName: \"kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.553095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.553081 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.654863 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.654777 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.655033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.654883 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9tx\" (UniqueName: \"kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.655033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.654982 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.655501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.655478 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.657459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.657435 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.713862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.713832 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9tx\" (UniqueName: \"kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx\") pod \"maas-api-revoke-sub-50b0a756-5xwl5\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:27.714064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:27.714048 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:28.104864 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:47:28.104826 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd249cc40_f495_4126_8a18_6d77d8b827e8.slice/crio-9a722edd8d1e9198e2465ff56b7992919d1a86f0b11baed71e3be9f05ccb292d WatchSource:0}: Error finding container 9a722edd8d1e9198e2465ff56b7992919d1a86f0b11baed71e3be9f05ccb292d: Status 404 returned error can't find the container with id 9a722edd8d1e9198e2465ff56b7992919d1a86f0b11baed71e3be9f05ccb292d Nov 02 17:47:28.105032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:28.104931 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5"] Nov 02 17:47:28.866988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:28.866945 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" event={"ID":"d249cc40-f495-4126-8a18-6d77d8b827e8","Type":"ContainerStarted","Data":"796e7e977f680bbf791c714652a8e5fb4cb9de0f53075e87ab49ca1db619574f"} Nov 02 17:47:28.867394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:28.866996 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" event={"ID":"d249cc40-f495-4126-8a18-6d77d8b827e8","Type":"ContainerStarted","Data":"9a722edd8d1e9198e2465ff56b7992919d1a86f0b11baed71e3be9f05ccb292d"} Nov 02 17:47:28.991169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:28.991119 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" podStartSLOduration=1.9911017229999999 podStartE2EDuration="1.991101723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:28.985874617 +0000 UTC m=+1436.059401820" watchObservedRunningTime="2026-11-02 17:47:28.991101723 +0000 UTC m=+1436.064628922" Nov 02 17:47:29.879069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:29.879018 5647 generic.go:356] "Generic (PLEG): container finished" podID="d249cc40-f495-4126-8a18-6d77d8b827e8" containerID="796e7e977f680bbf791c714652a8e5fb4cb9de0f53075e87ab49ca1db619574f" exitCode=0 Nov 02 17:47:29.879475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:29.879096 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" event={"ID":"d249cc40-f495-4126-8a18-6d77d8b827e8","Type":"ContainerDied","Data":"796e7e977f680bbf791c714652a8e5fb4cb9de0f53075e87ab49ca1db619574f"} Nov 02 17:47:31.030099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.030075 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:31.215485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.215405 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca\") pod \"d249cc40-f495-4126-8a18-6d77d8b827e8\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " Nov 02 17:47:31.215485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.215452 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx\" (UniqueName: \"kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx\") pod \"d249cc40-f495-4126-8a18-6d77d8b827e8\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " Nov 02 17:47:31.215683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.215666 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca\") pod \"d249cc40-f495-4126-8a18-6d77d8b827e8\" (UID: \"d249cc40-f495-4126-8a18-6d77d8b827e8\") " Nov 02 17:47:31.215804 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.215783 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca" pod "d249cc40-f495-4126-8a18-6d77d8b827e8" (UID: "d249cc40-f495-4126-8a18-6d77d8b827e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:31.216314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.216288 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:31.217779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.217751 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx" pod "d249cc40-f495-4126-8a18-6d77d8b827e8" (UID: "d249cc40-f495-4126-8a18-6d77d8b827e8"). InnerVolumeSpecName "kube-api-access-gz9tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:31.217866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.217780 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca" pod "d249cc40-f495-4126-8a18-6d77d8b827e8" (UID: "d249cc40-f495-4126-8a18-6d77d8b827e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:31.317552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.317510 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d249cc40-f495-4126-8a18-6d77d8b827e8-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:31.317552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.317545 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9tx\" (UniqueName: \"kubernetes.io/projected/d249cc40-f495-4126-8a18-6d77d8b827e8-kube-api-access-gz9tx\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:31.894418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.894384 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" Nov 02 17:47:31.894607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.894422 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5" event={"ID":"d249cc40-f495-4126-8a18-6d77d8b827e8","Type":"ContainerDied","Data":"9a722edd8d1e9198e2465ff56b7992919d1a86f0b11baed71e3be9f05ccb292d"} Nov 02 17:47:31.894607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:31.894481 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a722edd8d1e9198e2465ff56b7992919d1a86f0b11baed71e3be9f05ccb292d" Nov 02 17:47:32.053005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:32.052968 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6"] Nov 02 17:47:32.076984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:32.076952 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cbe9572-bhxp6"] Nov 02 17:47:33.082347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:33.082312 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32877e8-d26f-48e5-975d-3c3c222c85bd" path="/var/lib/kubelet/pods/e32877e8-d26f-48e5-975d-3c3c222c85bd/volumes" Nov 02 17:47:34.050099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.050062 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz"] Nov 02 17:47:34.100774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.100727 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1160493-qf4jz"] Nov 02 17:47:34.727631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.727597 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7529-356ae3"] Nov 02 17:47:34.728280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.728264 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d249cc40-f495-4126-8a18-6d77d8b827e8" containerName="revoke-keys" Nov 02 17:47:34.728338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.728282 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d249cc40-f495-4126-8a18-6d77d8b827e8" containerName="revoke-keys" Nov 02 17:47:34.728455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.728447 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d249cc40-f495-4126-8a18-6d77d8b827e8" containerName="revoke-keys" Nov 02 17:47:34.732244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.732229 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:34.765135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.765100 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkmd5\" (UniqueName: \"kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5\") pod \"test-curl-7529-356ae3\" (UID: \"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6\") " pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:34.783938 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.783913 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-356ae3"] Nov 02 17:47:34.830444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.830416 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4gwc\"" Nov 02 17:47:34.866555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.866526 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkmd5\" (UniqueName: \"kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5\") pod \"test-curl-7529-356ae3\" (UID: \"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6\") " pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:34.925431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:34.925403 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkmd5\" (UniqueName: \"kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5\") pod \"test-curl-7529-356ae3\" (UID: \"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6\") " pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:35.045036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:35.044947 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:35.083332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:35.083300 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed45447-9b90-42b6-ab6a-aa3b41153882" path="/var/lib/kubelet/pods/3ed45447-9b90-42b6-ab6a-aa3b41153882/volumes" Nov 02 17:47:35.427373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:35.427323 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-356ae3"] Nov 02 17:47:35.427833 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:47:35.427474 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65326e8c_790a_4ea7_8914_0dcfb2d9b0c6.slice/crio-bf200d65784827e3b467eddb72db35ef8f406d1b38706831b5c472464ba2a064 WatchSource:0}: Error finding container bf200d65784827e3b467eddb72db35ef8f406d1b38706831b5c472464ba2a064: Status 404 returned error can't find the container with id bf200d65784827e3b467eddb72db35ef8f406d1b38706831b5c472464ba2a064 Nov 02 17:47:35.429531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:35.429517 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:47:35.925639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:35.925604 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-356ae3" event={"ID":"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6","Type":"ContainerStarted","Data":"bf200d65784827e3b467eddb72db35ef8f406d1b38706831b5c472464ba2a064"} Nov 02 17:47:35.947533 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:35.947497 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:47:39.956649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:39.956614 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-356ae3" event={"ID":"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6","Type":"ContainerStarted","Data":"520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a"} Nov 02 17:47:40.075197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:40.074557 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7529-356ae3" podStartSLOduration=2.507866137 podStartE2EDuration="6.074537859s" totalImagesPullingTime="3.566671722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:40.069274076 +0000 UTC m=+1447.142801279" watchObservedRunningTime="2026-11-02 17:47:40.074537859 +0000 UTC m=+1447.148065062" Nov 02 17:47:41.566051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:41.565799 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7529-356ae3"] Nov 02 17:47:41.571526 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:41.571493 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7529-356ae3"] Nov 02 17:47:41.971075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:41.971019 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7529-356ae3" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" containerName="test-curl-7529-356ae3" containerID="cri-o://520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a" gracePeriod=2 Nov 02 17:47:42.023233 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:42.023194 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-356ae3\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:42.466045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.466004 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx"] Nov 02 17:47:42.467036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.467000 5647 cpu_manager.go:414] "containerMap: removing container" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" containerName="test-curl-7529-356ae3" Nov 02 17:47:42.467036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.467028 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" containerName="test-curl-7529-356ae3" Nov 02 17:47:42.467256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.467210 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" containerName="test-curl-7529-356ae3" Nov 02 17:47:42.471818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.471793 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.522270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.522231 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx"] Nov 02 17:47:42.565979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.565946 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:47:42.566391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.566375 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:47:42.612081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.612042 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.620835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.619956 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.620835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.620333 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.653757 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:42.650070 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-356ae3\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:42.726813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.724644 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.726813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.724805 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.726813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.725050 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.726813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.726607 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.743755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.739368 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.794355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.794318 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp\") pod \"maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:42.978521 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.978442 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz"] Nov 02 17:47:42.982890 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:42.982872 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.035478 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:43.035429 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-356ae3\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:43.073615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.073587 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e61288-openshift-default-dockercfg-lhxhc\"" Nov 02 17:47:43.088945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.088912 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:43.099230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.099183 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz"] Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.153770 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.153824 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.153883 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.153939 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.153972 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.154042 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.154140 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.154186 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.161769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.154270 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.259605 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.259510 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.261486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.261457 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.261707 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.261689 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.261855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.261812 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.262134 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.262115 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.262798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.262383 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.262996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.262958 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.263217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.263176 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.263394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.263378 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.263551 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.263536 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.264095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.262449 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.264524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.264012 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.264696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.261813 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.264831 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.264230 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.267721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.266609 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.267721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.266640 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.331069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.330766 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.331069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.331018 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token\") pod \"e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.608771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.605821 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:43.667677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.667642 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx"] Nov 02 17:47:43.696419 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:47:43.696386 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf5e7f_e4e9_421f_aada_6ecef248907e.slice/crio-624fc11f706f71618a4158f7596d63eba9e045706d0a533da249d1c90d7ca053 WatchSource:0}: Error finding container 624fc11f706f71618a4158f7596d63eba9e045706d0a533da249d1c90d7ca053: Status 404 returned error can't find the container with id 624fc11f706f71618a4158f7596d63eba9e045706d0a533da249d1c90d7ca053 Nov 02 17:47:43.986717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:43.986688 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" event={"ID":"0baf5e7f-e4e9-421f-aada-6ecef248907e","Type":"ContainerStarted","Data":"624fc11f706f71618a4158f7596d63eba9e045706d0a533da249d1c90d7ca053"} Nov 02 17:47:44.049053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.049013 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9"] Nov 02 17:47:44.054991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.054934 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c21765ce-g5fx9"] Nov 02 17:47:44.130426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.130315 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz"] Nov 02 17:47:44.131080 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:47:44.131056 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfaf2ab1_8a7a_45cb_8741_f8e266a87751.slice/crio-2c8f9d0540f5dc44c000f034be65c05d2cebb96064ccde76f6696522538cbdcb WatchSource:0}: Error finding container 2c8f9d0540f5dc44c000f034be65c05d2cebb96064ccde76f6696522538cbdcb: Status 404 returned error can't find the container with id 2c8f9d0540f5dc44c000f034be65c05d2cebb96064ccde76f6696522538cbdcb Nov 02 17:47:44.134447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.134410 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:47:44.134525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.134485 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:47:44.134525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.134515 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:47:44.254693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.254671 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:44.306847 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:44.306804 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-356ae3\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:44.319664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.319631 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5\" (UniqueName: \"kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5\") pod \"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6\" (UID: \"65326e8c-790a-4ea7-8914-0dcfb2d9b0c6\") " Nov 02 17:47:44.321924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.321897 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5" pod "65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" (UID: "65326e8c-790a-4ea7-8914-0dcfb2d9b0c6"). InnerVolumeSpecName "kube-api-access-fkmd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:44.422265 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.422227 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkmd5\" (UniqueName: \"kubernetes.io/projected/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6-kube-api-access-fkmd5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:44.995635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.995545 5647 generic.go:356] "Generic (PLEG): container finished" podID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" containerID="520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a" exitCode=137 Nov 02 17:47:44.995635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.995601 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:44.996203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.995647 5647 scope.go:122] "RemoveContainer" containerID="520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a" Nov 02 17:47:44.997424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.997399 5647 generic.go:356] "Generic (PLEG): container finished" podID="0baf5e7f-e4e9-421f-aada-6ecef248907e" containerID="7eff6cce986137c99379ebe87818100a256af1b8f92083420477ac33ab61efcf" exitCode=0 Nov 02 17:47:44.997558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.997480 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" event={"ID":"0baf5e7f-e4e9-421f-aada-6ecef248907e","Type":"ContainerDied","Data":"7eff6cce986137c99379ebe87818100a256af1b8f92083420477ac33ab61efcf"} Nov 02 17:47:44.999366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.999344 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" event={"ID":"bfaf2ab1-8a7a-45cb-8741-f8e266a87751","Type":"ContainerStarted","Data":"44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c"} Nov 02 17:47:44.999487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:44.999386 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" event={"ID":"bfaf2ab1-8a7a-45cb-8741-f8e266a87751","Type":"ContainerStarted","Data":"2c8f9d0540f5dc44c000f034be65c05d2cebb96064ccde76f6696522538cbdcb"} Nov 02 17:47:45.006561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.006538 5647 scope.go:122] "RemoveContainer" containerID="520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a" Nov 02 17:47:45.006991 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:45.006970 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a\": container with ID starting with 520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a not found: ID does not exist" containerID="520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a" Nov 02 17:47:45.007069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.007000 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a"} err="failed to get container status \"520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a\": rpc error: code = NotFound desc = could not find container \"520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a\": container with ID starting with 520c172acc4190dc3086577c8467f39a2fe6adb8da96775d24c40b7a23aebf3a not found: ID does not exist" Nov 02 17:47:45.047844 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:45.047807 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-356ae3\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:45.084336 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.084300 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" path="/var/lib/kubelet/pods/65326e8c-790a-4ea7-8914-0dcfb2d9b0c6/volumes" Nov 02 17:47:45.084957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.084940 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53432b2-16cb-4a66-a4d8-c24101062f1b" path="/var/lib/kubelet/pods/c53432b2-16cb-4a66-a4d8-c24101062f1b/volumes" Nov 02 17:47:45.217575 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:47:45.217533 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-356ae3\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="65326e8c-790a-4ea7-8914-0dcfb2d9b0c6" pod="opendatahub/test-curl-7529-356ae3" Nov 02 17:47:45.339213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.339158 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podStartSLOduration=3.339143359 podStartE2EDuration="3.339143359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:45.336324764 +0000 UTC m=+1452.409852005" watchObservedRunningTime="2026-11-02 17:47:45.339143359 +0000 UTC m=+1452.412670562" Nov 02 17:47:45.609208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.609131 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:45.614365 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:45.614341 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:46.008658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.008571 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:46.009707 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.009685 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:47:46.197298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.197262 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:46.249086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.249032 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca\") pod \"0baf5e7f-e4e9-421f-aada-6ecef248907e\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " Nov 02 17:47:46.250273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.250218 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca" pod "0baf5e7f-e4e9-421f-aada-6ecef248907e" (UID: "0baf5e7f-e4e9-421f-aada-6ecef248907e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:47:46.251745 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.251699 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca\") pod \"0baf5e7f-e4e9-421f-aada-6ecef248907e\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " Nov 02 17:47:46.251971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.251956 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp\") pod \"0baf5e7f-e4e9-421f-aada-6ecef248907e\" (UID: \"0baf5e7f-e4e9-421f-aada-6ecef248907e\") " Nov 02 17:47:46.253663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.253639 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:46.255925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.255899 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca" pod "0baf5e7f-e4e9-421f-aada-6ecef248907e" (UID: "0baf5e7f-e4e9-421f-aada-6ecef248907e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:47:46.256037 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.255967 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp" pod "0baf5e7f-e4e9-421f-aada-6ecef248907e" (UID: "0baf5e7f-e4e9-421f-aada-6ecef248907e"). InnerVolumeSpecName "kube-api-access-lmthp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:47:46.355719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.355664 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0baf5e7f-e4e9-421f-aada-6ecef248907e-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:46.355719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:46.355706 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmthp\" (UniqueName: \"kubernetes.io/projected/0baf5e7f-e4e9-421f-aada-6ecef248907e-kube-api-access-lmthp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:47:47.017007 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:47.016971 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" Nov 02 17:47:47.017007 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:47.017003 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx" event={"ID":"0baf5e7f-e4e9-421f-aada-6ecef248907e","Type":"ContainerDied","Data":"624fc11f706f71618a4158f7596d63eba9e045706d0a533da249d1c90d7ca053"} Nov 02 17:47:47.017501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:47.017050 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="624fc11f706f71618a4158f7596d63eba9e045706d0a533da249d1c90d7ca053" Nov 02 17:47:48.046784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:48.046751 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn"] Nov 02 17:47:48.097324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:48.097282 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d9bf31-gsktn"] Nov 02 17:47:49.082036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:49.081992 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e5fb23-0d06-4ff0-a202-072325c055f0" path="/var/lib/kubelet/pods/19e5fb23-0d06-4ff0-a202-072325c055f0/volumes" Nov 02 17:47:51.808650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.805551 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn"] Nov 02 17:47:51.808650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.806825 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0baf5e7f-e4e9-421f-aada-6ecef248907e" containerName="revoke-keys" Nov 02 17:47:51.808650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.806847 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baf5e7f-e4e9-421f-aada-6ecef248907e" containerName="revoke-keys" Nov 02 17:47:51.808650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.807080 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baf5e7f-e4e9-421f-aada-6ecef248907e" containerName="revoke-keys" Nov 02 17:47:51.829042 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.829005 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn"] Nov 02 17:47:51.829354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.829337 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:51.927313 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.923087 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l"] Nov 02 17:47:51.930901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:51.930872 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.116145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.072955 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e61288\"" Nov 02 17:47:52.117271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.117238 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e61288-dockercfg-45472\"" Nov 02 17:47:52.167107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.166901 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.167492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.167471 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwqr\" (UniqueName: \"kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr\") pod \"payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.167853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.167830 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25g6b\" (UniqueName: \"kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b\") pod \"payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.168058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.168039 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.176717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.175670 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l"] Nov 02 17:47:52.271016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.270973 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.271234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.271187 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.271314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.271301 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwqr\" (UniqueName: \"kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr\") pod \"payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.271378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.271356 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25g6b\" (UniqueName: \"kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b\") pod \"payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.271877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.271853 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.271993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.271971 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.328969 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.328933 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25g6b\" (UniqueName: \"kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b\") pod \"payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.329135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.328972 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwqr\" (UniqueName: \"kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr\") pod \"payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.473869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.473777 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:52.488893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.488863 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:52.860492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.860461 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn"] Nov 02 17:47:52.900116 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:47:52.900082 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27198416_6510_478e_b361_855d53ff3e40.slice/crio-fac44be08f5a5da2dca7017cd4479e87569653a4bfc10ffc54761824ff736872 WatchSource:0}: Error finding container fac44be08f5a5da2dca7017cd4479e87569653a4bfc10ffc54761824ff736872: Status 404 returned error can't find the container with id fac44be08f5a5da2dca7017cd4479e87569653a4bfc10ffc54761824ff736872 Nov 02 17:47:52.912030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:52.911999 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l"] Nov 02 17:47:52.915116 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:47:52.915090 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39e0598_ceb8_4d1b_b992_006710906761.slice/crio-df08db65211c391c3fa1143b0bfe7995cb8b12e03665a3a43cdfc40e4806f451 WatchSource:0}: Error finding container df08db65211c391c3fa1143b0bfe7995cb8b12e03665a3a43cdfc40e4806f451: Status 404 returned error can't find the container with id df08db65211c391c3fa1143b0bfe7995cb8b12e03665a3a43cdfc40e4806f451 Nov 02 17:47:53.196522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.196383 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" event={"ID":"d39e0598-ceb8-4d1b-b992-006710906761","Type":"ContainerStarted","Data":"85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37"} Nov 02 17:47:53.196522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.196459 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" event={"ID":"d39e0598-ceb8-4d1b-b992-006710906761","Type":"ContainerStarted","Data":"df08db65211c391c3fa1143b0bfe7995cb8b12e03665a3a43cdfc40e4806f451"} Nov 02 17:47:53.196522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.196492 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:47:53.198206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.198182 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" event={"ID":"27198416-6510-478e-b361-855d53ff3e40","Type":"ContainerStarted","Data":"4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5"} Nov 02 17:47:53.198308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.198223 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" event={"ID":"27198416-6510-478e-b361-855d53ff3e40","Type":"ContainerStarted","Data":"fac44be08f5a5da2dca7017cd4479e87569653a4bfc10ffc54761824ff736872"} Nov 02 17:47:53.198308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.198246 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:47:53.361769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.360900 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" podStartSLOduration=2.3608755280000002 podStartE2EDuration="2.360875528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:53.337670472 +0000 UTC m=+1460.411197675" watchObservedRunningTime="2026-11-02 17:47:53.360875528 +0000 UTC m=+1460.434402733" Nov 02 17:47:53.480833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.475602 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" podStartSLOduration=2.475582112 podStartE2EDuration="2.475582112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:53.466041961 +0000 UTC m=+1460.539569193" watchObservedRunningTime="2026-11-02 17:47:53.475582112 +0000 UTC m=+1460.549109315" Nov 02 17:47:53.550922 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:47:53.547518 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:48:00.072757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:00.072057 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h"] Nov 02 17:48:00.105777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:00.103477 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fbf48cb-2st6h"] Nov 02 17:48:01.082917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:01.082834 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16510d45-1769-4918-9c36-be628783bfeb" path="/var/lib/kubelet/pods/16510d45-1769-4918-9c36-be628783bfeb/volumes" Nov 02 17:48:04.070761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.070310 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc"] Nov 02 17:48:04.128760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.128210 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1638eb12-j8xwc"] Nov 02 17:48:04.214654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.214616 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:48:04.217335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.217304 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:48:04.432111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.432020 5647 generic.go:356] "Generic (PLEG): container finished" podID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerID="201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2" exitCode=0 Nov 02 17:48:04.432297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.432173 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" event={"ID":"e6a491df-b1ac-4bd6-9a99-e114a01c35dc","Type":"ContainerDied","Data":"201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2"} Nov 02 17:48:04.432626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.432609 5647 scope.go:122] "RemoveContainer" containerID="201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2" Nov 02 17:48:04.434230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.434201 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:04.434286 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.434272 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:04.434326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:04.434301 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:05.097761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:05.093140 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13333e50-d39a-4fbe-90d9-c67ed0e21139" path="/var/lib/kubelet/pods/13333e50-d39a-4fbe-90d9-c67ed0e21139/volumes" Nov 02 17:48:05.459566 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:05.459482 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" event={"ID":"e6a491df-b1ac-4bd6-9a99-e114a01c35dc","Type":"ContainerStarted","Data":"ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb"} Nov 02 17:48:05.976223 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:48:05.976182 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:48:06.172033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.172000 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:48:06.172439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.172103 5647 patch_prober.go:29] interesting pod/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:48:06.172439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.172139 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:48:06.671131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.671098 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv"] Nov 02 17:48:06.674427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.674409 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.758353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.758303 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv"] Nov 02 17:48:06.780143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.780109 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:48:06.780318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.780153 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:48:06.782567 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.782546 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.782631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.782612 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpcsv\" (UniqueName: \"kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.782690 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.782679 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.883726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.883690 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpcsv\" (UniqueName: \"kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.883965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.883784 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.884053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.883970 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.884488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.884464 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.886438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.886420 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.955274 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.955196 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpcsv\" (UniqueName: \"kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv\") pod \"maas-api-revoke-sub-125fbf55-5hgsv\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:06.987374 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:06.987337 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:07.176757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:07.174141 5647 patch_prober.go:29] interesting pod/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:48:07.176757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:07.174250 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:48:07.579493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:07.579463 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv"] Nov 02 17:48:07.579724 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:48:07.579691 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e4dffac_203a_422d_a4f8_a3a75ced37e3.slice/crio-da4f0bbab38ae56f7ae18a1fa7749a681b0fa12d974cc53b9ff7c86a2cd5565c WatchSource:0}: Error finding container da4f0bbab38ae56f7ae18a1fa7749a681b0fa12d974cc53b9ff7c86a2cd5565c: Status 404 returned error can't find the container with id da4f0bbab38ae56f7ae18a1fa7749a681b0fa12d974cc53b9ff7c86a2cd5565c Nov 02 17:48:08.171216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:08.171184 5647 patch_prober.go:29] interesting pod/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:48:08.171330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:08.171232 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:48:08.484642 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:08.484608 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" event={"ID":"6e4dffac-203a-422d-a4f8-a3a75ced37e3","Type":"ContainerStarted","Data":"5660a09db49b9de8930c2fb7263b3cc03631c05b86fef930c811d09ef8e340f5"} Nov 02 17:48:08.485064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:08.484685 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" event={"ID":"6e4dffac-203a-422d-a4f8-a3a75ced37e3","Type":"ContainerStarted","Data":"da4f0bbab38ae56f7ae18a1fa7749a681b0fa12d974cc53b9ff7c86a2cd5565c"} Nov 02 17:48:08.637761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:08.637577 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" podStartSLOduration=2.63755412 podStartE2EDuration="2.63755412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:08.610563204 +0000 UTC m=+1475.684090406" watchObservedRunningTime="2026-11-02 17:48:08.63755412 +0000 UTC m=+1475.711081323" Nov 02 17:48:09.171132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:09.171098 5647 patch_prober.go:29] interesting pod/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:48:09.171308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:09.171155 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:48:09.494412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:09.494331 5647 generic.go:356] "Generic (PLEG): container finished" podID="6e4dffac-203a-422d-a4f8-a3a75ced37e3" containerID="5660a09db49b9de8930c2fb7263b3cc03631c05b86fef930c811d09ef8e340f5" exitCode=0 Nov 02 17:48:09.494802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:09.494409 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" event={"ID":"6e4dffac-203a-422d-a4f8-a3a75ced37e3","Type":"ContainerDied","Data":"5660a09db49b9de8930c2fb7263b3cc03631c05b86fef930c811d09ef8e340f5"} Nov 02 17:48:10.171958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.171920 5647 patch_prober.go:29] interesting pod/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 02 17:48:10.172150 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.171985 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 02 17:48:10.716226 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.716198 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:10.859028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.858978 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca\") pod \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " Nov 02 17:48:10.859238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.859057 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv\" (UniqueName: \"kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv\") pod \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " Nov 02 17:48:10.859238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.859167 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca\") pod \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\" (UID: \"6e4dffac-203a-422d-a4f8-a3a75ced37e3\") " Nov 02 17:48:10.860170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.860136 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca" pod "6e4dffac-203a-422d-a4f8-a3a75ced37e3" (UID: "6e4dffac-203a-422d-a4f8-a3a75ced37e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:10.862821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.862778 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca" pod "6e4dffac-203a-422d-a4f8-a3a75ced37e3" (UID: "6e4dffac-203a-422d-a4f8-a3a75ced37e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:10.864213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.864189 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv" pod "6e4dffac-203a-422d-a4f8-a3a75ced37e3" (UID: "6e4dffac-203a-422d-a4f8-a3a75ced37e3"). InnerVolumeSpecName "kube-api-access-fpcsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:10.961463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.961420 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:10.961463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.961468 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpcsv\" (UniqueName: \"kubernetes.io/projected/6e4dffac-203a-422d-a4f8-a3a75ced37e3-kube-api-access-fpcsv\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:10.961679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:10.961485 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e4dffac-203a-422d-a4f8-a3a75ced37e3-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:11.176853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:11.176722 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:48:11.177117 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:11.177095 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:48:11.177918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:11.177885 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:48:11.547110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:11.547021 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" Nov 02 17:48:11.547110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:11.547048 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv" event={"ID":"6e4dffac-203a-422d-a4f8-a3a75ced37e3","Type":"ContainerDied","Data":"da4f0bbab38ae56f7ae18a1fa7749a681b0fa12d974cc53b9ff7c86a2cd5565c"} Nov 02 17:48:11.547355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:11.547124 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4f0bbab38ae56f7ae18a1fa7749a681b0fa12d974cc53b9ff7c86a2cd5565c" Nov 02 17:48:13.847191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:13.845883 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:50746->10.134.0.115:15021: read: connection reset by peer" start-of-body= Nov 02 17:48:13.847191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:13.845958 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": read tcp 10.134.0.2:50746->10.134.0.115:15021: read: connection reset by peer" Nov 02 17:48:14.574366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:14.574331 5647 generic.go:356] "Generic (PLEG): container finished" podID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerID="034d94b5082456eb0b87b52ade49347a98c6a6b48b3fb515eaaf9f6557e23a4a" exitCode=0 Nov 02 17:48:14.574568 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:14.574400 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" event={"ID":"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a","Type":"ContainerDied","Data":"034d94b5082456eb0b87b52ade49347a98c6a6b48b3fb515eaaf9f6557e23a4a"} Nov 02 17:48:14.574948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:14.574929 5647 scope.go:122] "RemoveContainer" containerID="034d94b5082456eb0b87b52ade49347a98c6a6b48b3fb515eaaf9f6557e23a4a" Nov 02 17:48:14.576700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:14.576665 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:14.576835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:14.576773 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:14.576835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:14.576810 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:15.047215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:15.047177 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k"] Nov 02 17:48:15.052577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:15.052547 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4e51282-zh64k"] Nov 02 17:48:15.093581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:15.093535 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2ea409-d287-47d4-b327-17cbaefe300c" path="/var/lib/kubelet/pods/0b2ea409-d287-47d4-b327-17cbaefe300c/volumes" Nov 02 17:48:15.586177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:15.586130 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" event={"ID":"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a","Type":"ContainerStarted","Data":"975f8aaf4169d02924c42c8f09c286bfaa5c02bd2e7fe45ed2921f61af61cbc9"} Nov 02 17:48:16.478775 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:16.478703 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:48:16.479373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:16.479039 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:48:16.479373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:16.479104 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:48:17.047282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:17.047225 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb"] Nov 02 17:48:17.097989 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:17.097959 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e111751c-t55rb"] Nov 02 17:48:17.478061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:17.478021 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:48:17.478276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:17.478074 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:48:18.051105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:18.051069 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87"] Nov 02 17:48:18.105784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:18.102056 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ce6a61-62m87"] Nov 02 17:48:18.478013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:18.477974 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:48:18.478194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:18.478034 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:48:19.481659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.481616 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:48:19.496210 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.496160 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:48:19.544790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.538630 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9135108c-a7a8-49bf-b084-585428f886c5" path="/var/lib/kubelet/pods/9135108c-a7a8-49bf-b084-585428f886c5/volumes" Nov 02 17:48:19.544790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.539591 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66ba264-a83a-4b03-b4a6-74fb21c29737" path="/var/lib/kubelet/pods/b66ba264-a83a-4b03-b4a6-74fb21c29737/volumes" Nov 02 17:48:19.894230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.879546 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs"] Nov 02 17:48:19.979145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.979112 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6e4dffac-203a-422d-a4f8-a3a75ced37e3" containerName="revoke-keys" Nov 02 17:48:19.979392 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.979375 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4dffac-203a-422d-a4f8-a3a75ced37e3" containerName="revoke-keys" Nov 02 17:48:19.979706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:19.979685 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4dffac-203a-422d-a4f8-a3a75ced37e3" containerName="revoke-keys" Nov 02 17:48:20.007827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.007791 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs"] Nov 02 17:48:20.008186 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.008167 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.101756 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.101821 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.101868 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.101892 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.101935 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.101982 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.102022 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbqk\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.102076 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.103761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.102125 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.160844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.146405 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f0f3e1-openshift-default-dockercfg-7h7sb\"" Nov 02 17:48:20.209946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.209883 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.210289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.210260 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.210435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.210417 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.210565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.210551 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.210709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.210695 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.211009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.210986 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbqk\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.211194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.211179 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.211345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.211329 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.211797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.211774 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.253763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.235602 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.253763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.235902 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.304593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.281155 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.304593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.281453 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.304593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.300848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.426447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.426329 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.431019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.430984 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.431845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.431819 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.469882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.458010 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbqk\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk\") pod \"e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:20.485441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.485409 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:48:20.486063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.486030 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:48:20.707510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:20.707119 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:21.285246 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.284077 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:21.390497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.390459 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw"] Nov 02 17:48:21.404070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.403990 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.488776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.476362 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 17:48:21.489328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.485341 5647 patch_prober.go:29] interesting pod/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Nov 02 17:48:21.489484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.489456 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Nov 02 17:48:21.489945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.485990 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 17:48:21.490671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.490643 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjfh\" (UniqueName: \"kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.490912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.490893 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.491125 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.491106 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.637689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.613568 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjfh\" (UniqueName: \"kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.637689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.613685 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.637689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.613802 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.637689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.615185 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.850945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.834513 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:21.850945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.834617 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:21.850945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.834670 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:21.850945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.835816 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.850945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.848546 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjfh\" (UniqueName: \"kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh\") pod \"maas-api-revoke-sub-4f56f9ac-2jhkw\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:21.862848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.862811 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw"] Nov 02 17:48:21.919185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:21.919141 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs"] Nov 02 17:48:22.011608 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:22.011504 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" event={"ID":"752613f5-4c69-4c73-a417-93325548fc69","Type":"ContainerStarted","Data":"0c65d3158ed113fcc1b5cc30d7952dd5ad21321b5f4a61edd8dea20ceb37d57b"} Nov 02 17:48:22.099117 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:22.099063 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:22.877148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:22.808080 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:48:22.877148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:22.809657 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:48:22.921357 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:22.917032 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:48:22.979315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:22.978971 5647 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": context deadline exceeded" Nov 02 17:48:23.404522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:23.403231 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 17:48:23.511337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:23.483518 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:23.512784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:23.512709 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:23.606104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:23.606055 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:23.751260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:23.723809 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:48:23.973763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:23.959370 5647 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Nov 02 17:48:24.055808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.052757 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" event={"ID":"752613f5-4c69-4c73-a417-93325548fc69","Type":"ContainerStarted","Data":"b3faf3307456bd8a2c1c578de252d61e01d452e45d55ddd16b256f80d1b688af"} Nov 02 17:48:24.055808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.052852 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw"] Nov 02 17:48:24.733051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.732961 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:24.738174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.738146 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:24.834432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.834387 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" event={"ID":"064e172b-a6a9-466c-a233-9ccd20ede71c","Type":"ContainerStarted","Data":"0f05b3bd0c1260d6f0904001984846536a1c4899403ce950a6bd3a75247b1c9a"} Nov 02 17:48:24.834621 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.834451 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" event={"ID":"064e172b-a6a9-466c-a233-9ccd20ede71c","Type":"ContainerStarted","Data":"0605e9d49fa62d1cad0fb774266c700901247312e3c24556ea784630cd5f9506"} Nov 02 17:48:24.834621 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.834601 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:24.835521 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.835502 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:48:24.857941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:24.857883 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podStartSLOduration=5.857868923 podStartE2EDuration="5.857868923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:24.231459264 +0000 UTC m=+1491.304986470" watchObservedRunningTime="2026-11-02 17:48:24.857868923 +0000 UTC m=+1491.931396123" Nov 02 17:48:25.105791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:25.102450 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" podStartSLOduration=4.102427951 podStartE2EDuration="4.102427951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:25.090647533 +0000 UTC m=+1492.164174737" watchObservedRunningTime="2026-11-02 17:48:25.102427951 +0000 UTC m=+1492.175955155" Nov 02 17:48:25.843185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:25.843151 5647 generic.go:356] "Generic (PLEG): container finished" podID="064e172b-a6a9-466c-a233-9ccd20ede71c" containerID="0f05b3bd0c1260d6f0904001984846536a1c4899403ce950a6bd3a75247b1c9a" exitCode=0 Nov 02 17:48:25.843389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:25.843244 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" event={"ID":"064e172b-a6a9-466c-a233-9ccd20ede71c","Type":"ContainerDied","Data":"0f05b3bd0c1260d6f0904001984846536a1c4899403ce950a6bd3a75247b1c9a"} Nov 02 17:48:27.077294 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.077262 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:27.131557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.131474 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh\" (UniqueName: \"kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh\") pod \"064e172b-a6a9-466c-a233-9ccd20ede71c\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " Nov 02 17:48:27.131760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.131690 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca\") pod \"064e172b-a6a9-466c-a233-9ccd20ede71c\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " Nov 02 17:48:27.131850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.131779 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca\") pod \"064e172b-a6a9-466c-a233-9ccd20ede71c\" (UID: \"064e172b-a6a9-466c-a233-9ccd20ede71c\") " Nov 02 17:48:27.132193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.132163 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca" pod "064e172b-a6a9-466c-a233-9ccd20ede71c" (UID: "064e172b-a6a9-466c-a233-9ccd20ede71c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:27.132393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.132375 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:27.133052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.133028 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb"] Nov 02 17:48:27.133656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.133640 5647 cpu_manager.go:414] "containerMap: removing container" podUID="064e172b-a6a9-466c-a233-9ccd20ede71c" containerName="revoke-keys" Nov 02 17:48:27.133656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.133656 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="064e172b-a6a9-466c-a233-9ccd20ede71c" containerName="revoke-keys" Nov 02 17:48:27.133866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.133823 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="064e172b-a6a9-466c-a233-9ccd20ede71c" containerName="revoke-keys" Nov 02 17:48:27.133928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.133868 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh" pod "064e172b-a6a9-466c-a233-9ccd20ede71c" (UID: "064e172b-a6a9-466c-a233-9ccd20ede71c"). InnerVolumeSpecName "kube-api-access-mrjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:27.134251 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.134223 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca" pod "064e172b-a6a9-466c-a233-9ccd20ede71c" (UID: "064e172b-a6a9-466c-a233-9ccd20ede71c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:48:27.137562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.137543 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.204276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.204241 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3fa615-openshift-default-dockercfg-8xhdl\"" Nov 02 17:48:27.237654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.237543 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.238016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.237988 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdt2\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.238297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.238273 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.238497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.238479 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.241953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.241926 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.281268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.281225 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.296947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.296909 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.297317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.297292 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.297526 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.297508 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.297853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.297832 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/064e172b-a6a9-466c-a233-9ccd20ede71c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:27.297972 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.297957 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrjfh\" (UniqueName: \"kubernetes.io/projected/064e172b-a6a9-466c-a233-9ccd20ede71c-kube-api-access-mrjfh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:27.419862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.419760 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.420122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.420102 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.420274 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.420259 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.420426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.420412 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.420575 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.420562 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.420684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.420672 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.420978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.420959 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.421326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.421147 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdt2\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.421577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.421558 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.440873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.422801 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.440873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.436615 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.441518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.441451 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.441979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.441928 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.442326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.442306 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.484785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.483841 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.484785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.484000 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb"] Nov 02 17:48:27.486548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.486489 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.489571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.489511 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.514100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.513550 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdt2\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2\") pod \"e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.514100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.513866 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:27.911760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.899143 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" event={"ID":"064e172b-a6a9-466c-a233-9ccd20ede71c","Type":"ContainerDied","Data":"0605e9d49fa62d1cad0fb774266c700901247312e3c24556ea784630cd5f9506"} Nov 02 17:48:27.911760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.899199 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0605e9d49fa62d1cad0fb774266c700901247312e3c24556ea784630cd5f9506" Nov 02 17:48:27.911760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:27.899248 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw" Nov 02 17:48:28.047118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.047069 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8"] Nov 02 17:48:28.074255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.074164 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df2e2ced-jc4p8"] Nov 02 17:48:28.354512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.354473 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb"] Nov 02 17:48:28.354888 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:48:28.354467 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f284f18_defa_49a4_8b36_52ff0ef8eb13.slice/crio-b38995921d45533fdac9e4b860828340998ded07ccba3d1548d7699a6d0fbe92 WatchSource:0}: Error finding container b38995921d45533fdac9e4b860828340998ded07ccba3d1548d7699a6d0fbe92: Status 404 returned error can't find the container with id b38995921d45533fdac9e4b860828340998ded07ccba3d1548d7699a6d0fbe92 Nov 02 17:48:28.357299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.357258 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:28.357374 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.357348 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:28.357411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.357394 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:48:28.909542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.909506 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" event={"ID":"3f284f18-defa-49a4-8b36-52ff0ef8eb13","Type":"ContainerStarted","Data":"7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f"} Nov 02 17:48:28.909726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:28.909564 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" event={"ID":"3f284f18-defa-49a4-8b36-52ff0ef8eb13","Type":"ContainerStarted","Data":"b38995921d45533fdac9e4b860828340998ded07ccba3d1548d7699a6d0fbe92"} Nov 02 17:48:29.031816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:29.031764 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podStartSLOduration=2.031748497 podStartE2EDuration="2.031748497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:29.030548941 +0000 UTC m=+1496.104076144" watchObservedRunningTime="2026-11-02 17:48:29.031748497 +0000 UTC m=+1496.105275694" Nov 02 17:48:29.082018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:29.081978 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6804dc1-e6a7-445e-a369-43cd784d64e6" path="/var/lib/kubelet/pods/a6804dc1-e6a7-445e-a369-43cd784d64e6/volumes" Nov 02 17:48:29.518308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:29.518274 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:29.524017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:29.523989 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:29.917273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:29.917237 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:29.918376 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:29.918356 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:48:34.064771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.062316 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:48:34.066884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.066853 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7"] Nov 02 17:48:34.083005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.082974 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:48:34.099956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.099920 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c305d2c-gsqt7"] Nov 02 17:48:34.843671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.834530 5647 scope.go:122] "RemoveContainer" containerID="7831274d9902b0db79421ed72505f5e719ed9c47f4807fbcc282090a83393b75" Nov 02 17:48:34.880866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.880841 5647 scope.go:122] "RemoveContainer" containerID="eda40cf6b1b845297b46d252b05df2378a35ce926920dc4856970daff9fe2471" Nov 02 17:48:34.892034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:34.892005 5647 scope.go:122] "RemoveContainer" containerID="0aeea195a2c4ccd847c155137f9cf15a4f45b7231244298dc1a66aaf79aa2e68" Nov 02 17:48:35.081591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:35.081557 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1116c286-a714-43f3-bc7a-99e8e19d7cfe" path="/var/lib/kubelet/pods/1116c286-a714-43f3-bc7a-99e8e19d7cfe/volumes" Nov 02 17:48:35.952511 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:48:35.952470 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:48:43.897470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:43.897435 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4"] Nov 02 17:48:43.898031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:43.897624 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" containerID="cri-o://77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143" gracePeriod=30 Nov 02 17:48:46.048317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:46.048276 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk"] Nov 02 17:48:46.098707 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:46.098669 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ecf78dd-zbdgk"] Nov 02 17:48:47.082448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:47.082410 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07b8156-7c4d-45e8-9fe3-137a3b386f8d" path="/var/lib/kubelet/pods/b07b8156-7c4d-45e8-9fe3-137a3b386f8d/volumes" Nov 02 17:48:48.049425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:48.049385 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj"] Nov 02 17:48:48.097850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:48.097814 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-423c9af2-hjgfj"] Nov 02 17:48:49.094067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.094028 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b58511-d066-45b6-825f-8a6ebbd4a989" path="/var/lib/kubelet/pods/a5b58511-d066-45b6-825f-8a6ebbd4a989/volumes" Nov 02 17:48:49.176905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.176839 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:48:49.226476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226448 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226524 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226543 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226614 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226685 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226717 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226785 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226820 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.226892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226854 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket\") pod \"430deb5a-0705-48e0-a500-2fa09af8a42d\" (UID: \"430deb5a-0705-48e0-a500-2fa09af8a42d\") " Nov 02 17:48:49.227147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226814 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:49.227147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.226928 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:49.227147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227110 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:49.227259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227188 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:49.227259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227207 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:48:49.227675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227596 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.227675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227617 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/430deb5a-0705-48e0-a500-2fa09af8a42d-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.227675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227627 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.227675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227637 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.227675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.227649 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.229021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.228994 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:48:49.229256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.229229 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "kube-api-access-2tg9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:49.229465 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.229448 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:48:49.229519 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.229446 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token" pod "430deb5a-0705-48e0-a500-2fa09af8a42d" (UID: "430deb5a-0705-48e0-a500-2fa09af8a42d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:48:49.324702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.324671 5647 generic.go:356] "Generic (PLEG): container finished" podID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerID="77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143" exitCode=0 Nov 02 17:48:49.324888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.324723 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" Nov 02 17:48:49.324888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.324764 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" event={"ID":"430deb5a-0705-48e0-a500-2fa09af8a42d","Type":"ContainerDied","Data":"77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143"} Nov 02 17:48:49.324888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.324821 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4" event={"ID":"430deb5a-0705-48e0-a500-2fa09af8a42d","Type":"ContainerDied","Data":"f00e43c9a1c506c721f422774c7caa6e75471192d53ced8e32de86590f641abd"} Nov 02 17:48:49.324888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.324850 5647 scope.go:122] "RemoveContainer" containerID="77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143" Nov 02 17:48:49.329549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.329525 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.329654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.329554 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.329654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.329571 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.329654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.329586 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tg9x\" (UniqueName: \"kubernetes.io/projected/430deb5a-0705-48e0-a500-2fa09af8a42d-kube-api-access-2tg9x\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:48:49.336471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.336456 5647 scope.go:122] "RemoveContainer" containerID="02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826" Nov 02 17:48:49.347522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.347500 5647 scope.go:122] "RemoveContainer" containerID="77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143" Nov 02 17:48:49.347920 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:48:49.347892 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143\": container with ID starting with 77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143 not found: ID does not exist" containerID="77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143" Nov 02 17:48:49.348035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.347935 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143"} err="failed to get container status \"77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143\": rpc error: code = NotFound desc = could not find container \"77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143\": container with ID starting with 77492fdaf44c3bfd27a8aa88e0830b16dad8df0bef44fca617fcd08bcbc66143 not found: ID does not exist" Nov 02 17:48:49.348035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.347954 5647 scope.go:122] "RemoveContainer" containerID="02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826" Nov 02 17:48:49.348265 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:48:49.348236 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826\": container with ID starting with 02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826 not found: ID does not exist" containerID="02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826" Nov 02 17:48:49.348359 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.348271 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826"} err="failed to get container status \"02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826\": rpc error: code = NotFound desc = could not find container \"02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826\": container with ID starting with 02dd700c9ed430108ee5e61ff6f5becc8f18c0d577d130302297fd689047d826 not found: ID does not exist" Nov 02 17:48:49.462211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.462133 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4"] Nov 02 17:48:49.465406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:49.465386 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f8d61e-openshift-default-568887976f-qn7q4"] Nov 02 17:48:51.082457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:51.082425 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" path="/var/lib/kubelet/pods/430deb5a-0705-48e0-a500-2fa09af8a42d/volumes" Nov 02 17:48:57.049506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:57.049463 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67"] Nov 02 17:48:57.099538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:57.099508 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b55dbfe-sbx67"] Nov 02 17:48:59.081488 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:48:59.081454 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb9f49f-3988-4692-86fe-e4ee96c5ec05" path="/var/lib/kubelet/pods/ecb9f49f-3988-4692-86fe-e4ee96c5ec05/volumes" Nov 02 17:49:00.862035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:00.861999 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:49:03.048201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:03.048163 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n"] Nov 02 17:49:03.098708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:03.098678 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n"] Nov 02 17:49:03.129650 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:03.129613 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-a8487438-wv88n\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="7743d48e-e421-44ee-952c-97424e208e4c" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8487438-wv88n" Nov 02 17:49:04.046848 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:04.046811 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z"] Nov 02 17:49:04.097601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:04.097570 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5aa62b97-2ht2z"] Nov 02 17:49:05.081860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:05.081828 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75c08b62-6cd8-48f5-9ad3-da80790757ef" path="/var/lib/kubelet/pods/75c08b62-6cd8-48f5-9ad3-da80790757ef/volumes" Nov 02 17:49:05.082327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:05.082303 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7743d48e-e421-44ee-952c-97424e208e4c" path="/var/lib/kubelet/pods/7743d48e-e421-44ee-952c-97424e208e4c/volumes" Nov 02 17:49:05.948033 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:05.947997 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:49:07.621012 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:07.620979 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf"] Nov 02 17:49:07.621559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:07.621198 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" containerID="cri-o://8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9" gracePeriod=30 Nov 02 17:49:12.892190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.892126 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:49:12.962564 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962526 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.962783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962575 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.962783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962630 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.962783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962675 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.962783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962719 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.962783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962764 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.963062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962794 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.963062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962979 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:12.963062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.962991 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:12.963062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.963001 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.963062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.963059 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy\") pod \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\" (UID: \"9b906cd3-3ec1-4228-97f6-5de992a1aa81\") " Nov 02 17:49:12.963301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.963113 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:12.963301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.963172 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:12.963476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.963451 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:12.964048 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.964022 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:12.964222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.964051 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:12.964222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.964067 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:12.964222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.964083 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:12.964222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.964098 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:12.965360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.965330 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "kube-api-access-9shcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:12.965477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.965373 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:12.965477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.965409 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:12.965560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:12.965546 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo" pod "9b906cd3-3ec1-4228-97f6-5de992a1aa81" (UID: "9b906cd3-3ec1-4228-97f6-5de992a1aa81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:13.049190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.049153 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq"] Nov 02 17:49:13.053032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.053012 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c142a4-qbsmq"] Nov 02 17:49:13.065950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.065909 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:13.065950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.065946 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:13.065950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.065962 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shcr\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-kube-api-access-9shcr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:13.066155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.065974 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b906cd3-3ec1-4228-97f6-5de992a1aa81-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:13.083493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.083465 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9101e4b5-2ab3-466d-bbcd-e238bb95e035" path="/var/lib/kubelet/pods/9101e4b5-2ab3-466d-bbcd-e238bb95e035/volumes" Nov 02 17:49:13.594836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.594793 5647 generic.go:356] "Generic (PLEG): container finished" podID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerID="8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9" exitCode=0 Nov 02 17:49:13.595018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.594843 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" Nov 02 17:49:13.595018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.594875 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" event={"ID":"9b906cd3-3ec1-4228-97f6-5de992a1aa81","Type":"ContainerDied","Data":"8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9"} Nov 02 17:49:13.595018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.594937 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf" event={"ID":"9b906cd3-3ec1-4228-97f6-5de992a1aa81","Type":"ContainerDied","Data":"c079868c2eaf78599472d8a1bb901bda194e3092f0a8421654b23d8aab4209c5"} Nov 02 17:49:13.595018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.594968 5647 scope.go:122] "RemoveContainer" containerID="8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9" Nov 02 17:49:13.605994 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.605975 5647 scope.go:122] "RemoveContainer" containerID="8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9" Nov 02 17:49:13.606306 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:13.606289 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9\": container with ID starting with 8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9 not found: ID does not exist" containerID="8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9" Nov 02 17:49:13.606356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.606314 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9"} err="failed to get container status \"8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9\": rpc error: code = NotFound desc = could not find container \"8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9\": container with ID starting with 8fabedf8a83a4018979ed2b357ba1474f1cf54b33178f1f61c74d8bcaea554e9 not found: ID does not exist" Nov 02 17:49:13.764857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.764821 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf"] Nov 02 17:49:13.815491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:13.815459 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-cxfwf"] Nov 02 17:49:15.082401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:15.082369 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" path="/var/lib/kubelet/pods/9b906cd3-3ec1-4228-97f6-5de992a1aa81/volumes" Nov 02 17:49:16.456770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:16.454709 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx"] Nov 02 17:49:16.506194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:16.506142 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8cbedb9f-26klx"] Nov 02 17:49:17.082962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:17.082926 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baf5e7f-e4e9-421f-aada-6ecef248907e" path="/var/lib/kubelet/pods/0baf5e7f-e4e9-421f-aada-6ecef248907e/volumes" Nov 02 17:49:24.048307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:24.048269 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424"] Nov 02 17:49:24.098951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:24.098914 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef26c8b-jr424"] Nov 02 17:49:25.081980 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:25.081944 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd7c4aa-9171-4a18-9eb1-209b1572a861" path="/var/lib/kubelet/pods/bfd7c4aa-9171-4a18-9eb1-209b1572a861/volumes" Nov 02 17:49:26.933593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.933558 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr"] Nov 02 17:49:26.934185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934168 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" Nov 02 17:49:26.934185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934185 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" Nov 02 17:49:26.934273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934207 5647 cpu_manager.go:414] "containerMap: removing container" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" Nov 02 17:49:26.934273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934213 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" Nov 02 17:49:26.934273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934227 5647 cpu_manager.go:414] "containerMap: removing container" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" Nov 02 17:49:26.934273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934232 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" Nov 02 17:49:26.934430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934334 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" Nov 02 17:49:26.934430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934345 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="430deb5a-0705-48e0-a500-2fa09af8a42d" containerName="istio-proxy" Nov 02 17:49:26.934430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.934357 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b906cd3-3ec1-4228-97f6-5de992a1aa81" containerName="istio-proxy" Nov 02 17:49:26.938186 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.938167 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:26.993956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:26.993918 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr"] Nov 02 17:49:27.035238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.035188 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.035443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.035358 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2kbt\" (UniqueName: \"kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.040811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.040783 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8cbedb9f\"" Nov 02 17:49:27.041013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.040928 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8cbedb9f-dockercfg-hxsc6\"" Nov 02 17:49:27.136880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.136841 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.137069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.136921 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2kbt\" (UniqueName: \"kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.137693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.137664 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.347152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.347092 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2kbt\" (UniqueName: \"kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt\") pod \"payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.552589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.552552 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:27.930239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:27.930210 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr"] Nov 02 17:49:27.931315 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:49:27.931274 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30e5bbd_9f3d_44db_9df3_233a53d92b01.slice/crio-1bec9200afdf5f2456936d16c639d795e8b88065fcb22d83fa529414d3669019 WatchSource:0}: Error finding container 1bec9200afdf5f2456936d16c639d795e8b88065fcb22d83fa529414d3669019: Status 404 returned error can't find the container with id 1bec9200afdf5f2456936d16c639d795e8b88065fcb22d83fa529414d3669019 Nov 02 17:49:28.783841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:28.783795 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" event={"ID":"a30e5bbd-9f3d-44db-9df3-233a53d92b01","Type":"ContainerStarted","Data":"90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33"} Nov 02 17:49:28.784276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:28.783868 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" event={"ID":"a30e5bbd-9f3d-44db-9df3-233a53d92b01","Type":"ContainerStarted","Data":"1bec9200afdf5f2456936d16c639d795e8b88065fcb22d83fa529414d3669019"} Nov 02 17:49:28.784276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:28.784007 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:28.902926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:28.902872 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" podStartSLOduration=2.902856841 podStartE2EDuration="2.902856841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:28.900521947 +0000 UTC m=+1555.974049151" watchObservedRunningTime="2026-11-02 17:49:28.902856841 +0000 UTC m=+1555.976384043" Nov 02 17:49:30.047111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.047073 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg"] Nov 02 17:49:30.098082 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.098045 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07d6463a-5t7fg"] Nov 02 17:49:30.873850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.873809 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd"] Nov 02 17:49:30.874095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.874066 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" containerID="cri-o://c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b" gracePeriod=30 Nov 02 17:49:30.926853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.926804 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p"] Nov 02 17:49:30.927095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.927063 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" containerID="cri-o://25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810" gracePeriod=30 Nov 02 17:49:30.931838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.931809 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg"] Nov 02 17:49:30.932106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:30.932071 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" containerID="cri-o://9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac" gracePeriod=30 Nov 02 17:49:31.081789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:31.081754 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a012f974-c290-4733-af00-1199d668e0ef" path="/var/lib/kubelet/pods/a012f974-c290-4733-af00-1199d668e0ef/volumes" Nov 02 17:49:34.939926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.939886 5647 scope.go:122] "RemoveContainer" containerID="cd81187ebd33d7dd1882edada5729e7f92fd23b945824fac56ea58579edee9d1" Nov 02 17:49:34.950285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.950264 5647 scope.go:122] "RemoveContainer" containerID="2f27d6aaa0152dccf40af3d03b2c363bf6ddf85cc127342cbc0beee69ebd7e72" Nov 02 17:49:34.959654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.959628 5647 scope.go:122] "RemoveContainer" containerID="a9f9fd5381aea9a8ec8db8f228ec92ea8044bb810a41d62e4b1d24f43a0ecc5a" Nov 02 17:49:34.969286 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.969262 5647 scope.go:122] "RemoveContainer" containerID="5bd1e7e14fb7dc8e8dd15204c60ca0807b4b4b04f845750d7e64989fb124ba5a" Nov 02 17:49:34.978824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.978804 5647 scope.go:122] "RemoveContainer" containerID="379d494c437f78c4aaa294f427a239d6779ce0bb5d11e0f593707359c4a42d61" Nov 02 17:49:34.987836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.987819 5647 scope.go:122] "RemoveContainer" containerID="4bba6696efed6c9d2b3310b67cee167cd370c72b942b4b54ce56e9c874dc6e08" Nov 02 17:49:34.996664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:34.996646 5647 scope.go:122] "RemoveContainer" containerID="8177f1c474a90292897cc1c78d6f538aea6989bbef8f5f8d0cfb90a995a46636" Nov 02 17:49:35.005774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:35.005723 5647 scope.go:122] "RemoveContainer" containerID="4b6ad7943b2408623d295138f12e734cfb9ce2e94dad8d58aa674c22e43cda38" Nov 02 17:49:35.014557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:35.014535 5647 scope.go:122] "RemoveContainer" containerID="e5e22328f987b2f97a24570c8454e25aee4ce561c9ec35630bf3e7a20d4dbb39" Nov 02 17:49:35.952433 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:35.952398 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:49:36.224434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.224407 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:49:36.266203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.266168 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:49:36.269664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.269639 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:49:36.321549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321516 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.321769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321562 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.321769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321605 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.321769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321632 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.321769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321669 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.321769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321717 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321783 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321813 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321856 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321889 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321930 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321968 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322001 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.321971 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.322074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322064 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322133 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322174 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322189 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322223 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322281 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322343 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322377 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322431 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322482 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert\") pod \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\" (UID: \"013678d9-8093-4f8d-ab06-cf5a7cfc46d0\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322516 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322561 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322639 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.322695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322692 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322279 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322763 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322882 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs\") pod \"829651da-23ec-4567-8b0d-58677008d0a3\" (UID: \"829651da-23ec-4567-8b0d-58677008d0a3\") " Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322914 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322947 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token\") pod \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\" (UID: \"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956\") " Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.322497 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.323200 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.323468 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.323588 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.323690 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.323717 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.323929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.323749 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324093 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324466 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324535 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324552 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324569 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324584 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.324610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324600 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324613 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324626 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324640 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324654 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324667 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324681 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324695 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324708 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324722 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/829651da-23ec-4567-8b0d-58677008d0a3-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.325076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.324953 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:36.326896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.326862 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.327004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.326900 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.327004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.326952 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:36.327107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327046 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:36.327160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327106 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:36.327160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327107 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj" pod "013678d9-8093-4f8d-ab06-cf5a7cfc46d0" (UID: "013678d9-8093-4f8d-ab06-cf5a7cfc46d0"). InnerVolumeSpecName "kube-api-access-qnchj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:36.327255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327161 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:36.327255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327219 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "kube-api-access-fxctz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:36.327504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327488 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:36.327670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327653 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token" pod "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" (UID: "bd0bdf34-2fb7-4820-a7d9-75ce6c57b956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:36.327708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.327657 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk" pod "829651da-23ec-4567-8b0d-58677008d0a3" (UID: "829651da-23ec-4567-8b0d-58677008d0a3"). InnerVolumeSpecName "kube-api-access-9n4tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:36.426424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426336 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/829651da-23ec-4567-8b0d-58677008d0a3-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426369 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426382 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnchj\" (UniqueName: \"kubernetes.io/projected/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-kube-api-access-qnchj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426398 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/829651da-23ec-4567-8b0d-58677008d0a3-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426410 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426422 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426436 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n4tk\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-kube-api-access-9n4tk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426448 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426459 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxctz\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-kube-api-access-fxctz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426471 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/829651da-23ec-4567-8b0d-58677008d0a3-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426484 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426496 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.426777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.426507 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/013678d9-8093-4f8d-ab06-cf5a7cfc46d0-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:36.909637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.907673 5647 generic.go:356] "Generic (PLEG): container finished" podID="829651da-23ec-4567-8b0d-58677008d0a3" containerID="25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810" exitCode=0 Nov 02 17:49:36.909637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.907772 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" Nov 02 17:49:36.909637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.907818 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" event={"ID":"829651da-23ec-4567-8b0d-58677008d0a3","Type":"ContainerDied","Data":"25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810"} Nov 02 17:49:36.909637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.907892 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p" event={"ID":"829651da-23ec-4567-8b0d-58677008d0a3","Type":"ContainerDied","Data":"f4cdabdea1d832d3ac9373fca8f99041faf13c786265b83c4f7d18c940299f52"} Nov 02 17:49:36.909637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.907930 5647 scope.go:122] "RemoveContainer" containerID="25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810" Nov 02 17:49:36.911860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.911706 5647 generic.go:356] "Generic (PLEG): container finished" podID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerID="c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b" exitCode=0 Nov 02 17:49:36.914101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.914027 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" event={"ID":"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956","Type":"ContainerDied","Data":"c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b"} Nov 02 17:49:36.914337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.914275 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" event={"ID":"bd0bdf34-2fb7-4820-a7d9-75ce6c57b956","Type":"ContainerDied","Data":"781828897a18d298c3f0f868c91888bf4a3f7811eba574feed5cce2f9a985047"} Nov 02 17:49:36.914602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.914552 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd" Nov 02 17:49:36.928868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.928235 5647 generic.go:356] "Generic (PLEG): container finished" podID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerID="9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac" exitCode=0 Nov 02 17:49:36.928868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.928375 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" Nov 02 17:49:36.928868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.928374 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" event={"ID":"013678d9-8093-4f8d-ab06-cf5a7cfc46d0","Type":"ContainerDied","Data":"9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac"} Nov 02 17:49:36.928868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.928527 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg" event={"ID":"013678d9-8093-4f8d-ab06-cf5a7cfc46d0","Type":"ContainerDied","Data":"faa0c2fa59613082046dab19342051579061a80352ed7b646b4c2142c2f8c458"} Nov 02 17:49:36.931328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.931301 5647 scope.go:122] "RemoveContainer" containerID="25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810" Nov 02 17:49:36.931695 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:36.931639 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810\": container with ID starting with 25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810 not found: ID does not exist" containerID="25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810" Nov 02 17:49:36.931766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.931710 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810"} err="failed to get container status \"25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810\": rpc error: code = NotFound desc = could not find container \"25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810\": container with ID starting with 25b55b9f6d682fe62594513b9c95e2df54d6de389c7f46f6206992eef6e2b810 not found: ID does not exist" Nov 02 17:49:36.931766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.931753 5647 scope.go:122] "RemoveContainer" containerID="c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b" Nov 02 17:49:36.943757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.943720 5647 scope.go:122] "RemoveContainer" containerID="c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b" Nov 02 17:49:36.944200 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:36.944179 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b\": container with ID starting with c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b not found: ID does not exist" containerID="c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b" Nov 02 17:49:36.944341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.944315 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b"} err="failed to get container status \"c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b\": rpc error: code = NotFound desc = could not find container \"c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b\": container with ID starting with c0b229d62fd2ab2b70e3fd9c7df37270ca97e49059f6fb084c712aacb942fb7b not found: ID does not exist" Nov 02 17:49:36.944440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.944430 5647 scope.go:122] "RemoveContainer" containerID="9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac" Nov 02 17:49:36.971972 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.971941 5647 scope.go:122] "RemoveContainer" containerID="9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac" Nov 02 17:49:36.973043 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:36.972992 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac\": container with ID starting with 9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac not found: ID does not exist" containerID="9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac" Nov 02 17:49:36.973202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:36.973041 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac"} err="failed to get container status \"9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac\": rpc error: code = NotFound desc = could not find container \"9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac\": container with ID starting with 9a9021981e1fd565abaf453e85855ea34298b0d28d7c3179e51f716b89278cac not found: ID does not exist" Nov 02 17:49:37.087287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:37.087260 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd"] Nov 02 17:49:37.138057 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:37.138019 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-8wnpd"] Nov 02 17:49:37.249764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:37.249562 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p"] Nov 02 17:49:37.253675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:37.253647 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-tkf5p"] Nov 02 17:49:37.385283 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:37.383635 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg"] Nov 02 17:49:37.387128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:37.387108 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-2fhhg"] Nov 02 17:49:39.081393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:39.081357 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" path="/var/lib/kubelet/pods/013678d9-8093-4f8d-ab06-cf5a7cfc46d0/volumes" Nov 02 17:49:39.082109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:39.082090 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829651da-23ec-4567-8b0d-58677008d0a3" path="/var/lib/kubelet/pods/829651da-23ec-4567-8b0d-58677008d0a3/volumes" Nov 02 17:49:39.082572 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:39.082553 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" path="/var/lib/kubelet/pods/bd0bdf34-2fb7-4820-a7d9-75ce6c57b956/volumes" Nov 02 17:49:39.793965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:39.793940 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:49:41.701800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.701637 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt"] Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717357 5647 cpu_manager.go:414] "containerMap: removing container" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717387 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717446 5647 cpu_manager.go:414] "containerMap: removing container" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717456 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717478 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717485 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717770 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd0bdf34-2fb7-4820-a7d9-75ce6c57b956" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717798 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="013678d9-8093-4f8d-ab06-cf5a7cfc46d0" containerName="istio-proxy" Nov 02 17:49:41.720758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.717815 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="829651da-23ec-4567-8b0d-58677008d0a3" containerName="istio-proxy" Nov 02 17:49:41.724758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.723690 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:41.780000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.779966 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-27ab38-openshift-default-dockercfg-lxw55\"" Nov 02 17:49:41.938570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:41.931982 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt"] Nov 02 17:49:42.105825 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.009137 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbf7j\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.175760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.164935 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.209374 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.209460 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.209541 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.209578 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.209653 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.209689 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.219774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.210385 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.311602 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.311669 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.311765 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.311895 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbf7j\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.312000 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.312035 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.312074 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.312141 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.312181 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.312673 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.313791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.313354 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.321765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.320069 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.321765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.320324 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.331643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.322445 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.343251 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.343124 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.343590 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.343570 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.385364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.385294 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbf7j\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.385887 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.385863 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token\") pod \"e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:42.658111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:42.658021 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:43.137080 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:43.137040 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt"] Nov 02 17:49:43.137481 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:49:43.137289 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5171dfce_2108_4f03_8b01_c0fd91b3e5a3.slice/crio-78506386c589aa48eea0358f62eccc6826a904bed359e0f4668f4734b859a2e6 WatchSource:0}: Error finding container 78506386c589aa48eea0358f62eccc6826a904bed359e0f4668f4734b859a2e6: Status 404 returned error can't find the container with id 78506386c589aa48eea0358f62eccc6826a904bed359e0f4668f4734b859a2e6 Nov 02 17:49:43.140864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:43.140787 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:49:43.141002 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:43.140913 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:49:43.141002 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:43.140957 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:49:43.142098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:43.142072 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6"] Nov 02 17:49:43.152083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:43.152052 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebce122c-fktj6"] Nov 02 17:49:44.054261 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.054218 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6"] Nov 02 17:49:44.105393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.105347 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c21d131-g64b6"] Nov 02 17:49:44.132839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.132793 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" event={"ID":"5171dfce-2108-4f03-8b01-c0fd91b3e5a3","Type":"ContainerStarted","Data":"b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca"} Nov 02 17:49:44.133046 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.132904 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" event={"ID":"5171dfce-2108-4f03-8b01-c0fd91b3e5a3","Type":"ContainerStarted","Data":"78506386c589aa48eea0358f62eccc6826a904bed359e0f4668f4734b859a2e6"} Nov 02 17:49:44.263880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.263820 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podStartSLOduration=3.263804263 podStartE2EDuration="3.263804263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:44.260054159 +0000 UTC m=+1571.333581365" watchObservedRunningTime="2026-11-02 17:49:44.263804263 +0000 UTC m=+1571.337331466" Nov 02 17:49:44.658817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.658776 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:44.665227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:44.665195 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:45.082478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:45.082406 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3cab67-aa08-467b-9c77-e6cfd96ee9c3" path="/var/lib/kubelet/pods/8a3cab67-aa08-467b-9c77-e6cfd96ee9c3/volumes" Nov 02 17:49:45.082905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:45.082890 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966e4d2f-6350-47b4-942c-8b66bc87fe3f" path="/var/lib/kubelet/pods/966e4d2f-6350-47b4-942c-8b66bc87fe3f/volumes" Nov 02 17:49:45.139399 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:45.139363 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:45.140303 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:45.140285 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:49:46.528926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:46.527415 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": read tcp 10.134.0.2:36532->10.134.0.120:15021: read: connection reset by peer" start-of-body= Nov 02 17:49:46.528926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:46.527468 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": read tcp 10.134.0.2:36532->10.134.0.120:15021: read: connection reset by peer" Nov 02 17:49:47.156223 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:47.156194 5647 generic.go:356] "Generic (PLEG): container finished" podID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerID="44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c" exitCode=0 Nov 02 17:49:47.156437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:47.156266 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" event={"ID":"bfaf2ab1-8a7a-45cb-8741-f8e266a87751","Type":"ContainerDied","Data":"44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c"} Nov 02 17:49:47.156829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:47.156802 5647 scope.go:122] "RemoveContainer" containerID="44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c" Nov 02 17:49:47.158344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:47.158311 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:49:47.158456 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:47.158398 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:49:47.158456 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:47.158451 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:49:48.187759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.187632 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" event={"ID":"bfaf2ab1-8a7a-45cb-8741-f8e266a87751","Type":"ContainerStarted","Data":"1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7"} Nov 02 17:49:48.620768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.615123 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:49:48.620768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.615284 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:49:48.620768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.615334 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:49:48.713771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.712171 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb"] Nov 02 17:49:48.722760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.722139 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" containerID="cri-o://e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e" gracePeriod=30 Nov 02 17:49:48.759761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.748395 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq"] Nov 02 17:49:48.759761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.748642 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" containerID="cri-o://150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5" gracePeriod=30 Nov 02 17:49:48.773918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.773877 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn"] Nov 02 17:49:48.774482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.774426 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" containerID="cri-o://f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016" gracePeriod=30 Nov 02 17:49:48.802809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.802767 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl"] Nov 02 17:49:48.816294 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:48.816248 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" containerID="cri-o://76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8" gracePeriod=30 Nov 02 17:49:49.620762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:49.612123 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:49:49.620762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:49.612188 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:49:50.700764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:50.638381 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:49:50.700764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:50.638478 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:49:50.979814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:50.978798 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw"] Nov 02 17:49:51.352080 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.352054 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.370574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.370544 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw"] Nov 02 17:49:51.370886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.370868 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn"] Nov 02 17:49:51.376416 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.376395 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn"] Nov 02 17:49:51.376586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.376520 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.453709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.453673 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.453930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.453830 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.453930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.453901 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fdbz\" (UniqueName: \"kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz\") pod \"payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.454011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.453958 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpt87\" (UniqueName: \"kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87\") pod \"payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.459244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.459226 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-27ab38-dockercfg-wkjkz\"" Nov 02 17:49:51.459322 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.459263 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-27ab38\"" Nov 02 17:49:51.555642 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.555588 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.555880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.555777 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.555880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.555860 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fdbz\" (UniqueName: \"kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz\") pod \"payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.556035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.555897 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpt87\" (UniqueName: \"kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87\") pod \"payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.556302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.556281 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.556695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.556670 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.628312 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.628220 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:49:51.628557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.628528 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:49:51.631833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.631345 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpt87\" (UniqueName: \"kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87\") pod \"payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:51.631833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.631796 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fdbz\" (UniqueName: \"kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz\") pod \"payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.682109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.682073 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:51.690635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:51.690610 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:52.096910 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.096884 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw"] Nov 02 17:49:52.098890 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:49:52.098866 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3d3f6e_9e53_43d9_9194_0cc639023629.slice/crio-48f5f4e04a3b66af7ae4401f51c55539a8ba4bab674c7a33d9316bebefb45db4 WatchSource:0}: Error finding container 48f5f4e04a3b66af7ae4401f51c55539a8ba4bab674c7a33d9316bebefb45db4: Status 404 returned error can't find the container with id 48f5f4e04a3b66af7ae4401f51c55539a8ba4bab674c7a33d9316bebefb45db4 Nov 02 17:49:52.146013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.145984 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn"] Nov 02 17:49:52.146155 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:49:52.146136 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68ce3d4_faf6_49c2_aa4e_b36b3fe1e50a.slice/crio-bf36f48c7c531e30b18c2e056bb654a08af330f5098fb3a07f67d949df4a09f7 WatchSource:0}: Error finding container bf36f48c7c531e30b18c2e056bb654a08af330f5098fb3a07f67d949df4a09f7: Status 404 returned error can't find the container with id bf36f48c7c531e30b18c2e056bb654a08af330f5098fb3a07f67d949df4a09f7 Nov 02 17:49:52.236773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.236690 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:49:52.236956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.236776 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:49:52.381094 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.380988 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" event={"ID":"ff3d3f6e-9e53-43d9-9194-0cc639023629","Type":"ContainerStarted","Data":"b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d"} Nov 02 17:49:52.381094 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.381044 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" event={"ID":"ff3d3f6e-9e53-43d9-9194-0cc639023629","Type":"ContainerStarted","Data":"48f5f4e04a3b66af7ae4401f51c55539a8ba4bab674c7a33d9316bebefb45db4"} Nov 02 17:49:52.381333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.381101 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:49:52.382877 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.382853 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" event={"ID":"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a","Type":"ContainerStarted","Data":"410dc93745329c90ac847124eceece0154ffbd50048784e0c2ada4574fdb5452"} Nov 02 17:49:52.383014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.382894 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" event={"ID":"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a","Type":"ContainerStarted","Data":"bf36f48c7c531e30b18c2e056bb654a08af330f5098fb3a07f67d949df4a09f7"} Nov 02 17:49:52.383014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.382925 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:49:52.526758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.525875 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" podStartSLOduration=2.525852641 podStartE2EDuration="2.525852641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:52.499991139 +0000 UTC m=+1579.573518340" watchObservedRunningTime="2026-11-02 17:49:52.525852641 +0000 UTC m=+1579.599379846" Nov 02 17:49:52.621759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.621100 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" podStartSLOduration=1.62107986 podStartE2EDuration="1.62107986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:52.61930827 +0000 UTC m=+1579.692835474" watchObservedRunningTime="2026-11-02 17:49:52.62107986 +0000 UTC m=+1579.694607064" Nov 02 17:49:52.625758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.623107 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:49:52.625758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:52.623164 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:49:53.615768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:53.615276 5647 patch_prober.go:29] interesting pod/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 02 17:49:53.615768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:53.615360 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 02 17:49:53.668040 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:53.668004 5647 patch_prober.go:29] interesting pod/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:49:53.668208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:53.668065 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:49:54.065434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.065405 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097192 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097261 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097295 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097404 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097439 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097498 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097625 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097692 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.097866 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo\") pod \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\" (UID: \"0349f5cd-00ad-4b9e-9be8-587f1ae34f60\") " Nov 02 17:49:54.098454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098061 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098509 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098525 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098695 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098716 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098751 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098964 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.099133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.098979 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.102187 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.102156 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "kube-api-access-pzdhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.102480 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.102453 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.102630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.102599 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:54.113648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.113583 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy" pod "0349f5cd-00ad-4b9e-9be8-587f1ae34f60" (UID: "0349f5cd-00ad-4b9e-9be8-587f1ae34f60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.173703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.173634 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:49:54.199823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.199209 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.200195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.200172 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.200440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.200409 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.200685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.200662 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.200875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.200849 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.201053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.201027 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.201386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.201343 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.201567 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.201549 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.201756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.201719 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo\") pod \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\" (UID: \"040bc6b3-7fd7-4545-86a9-439ef63e92b0\") " Nov 02 17:49:54.204079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.203123 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.207989 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.207884 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzdhj\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-kube-api-access-pzdhj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.211477 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.211458 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.211685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.211669 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.211932 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.211888 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.212091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.212077 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349f5cd-00ad-4b9e-9be8-587f1ae34f60-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.212222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.205116 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.212341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.206190 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.212471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.206905 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.212578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.207260 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.212683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.207803 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:54.234322 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.234203 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "kube-api-access-w6dlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.234322 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.234201 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.234322 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.234289 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.236309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.236279 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo" pod "040bc6b3-7fd7-4545-86a9-439ef63e92b0" (UID: "040bc6b3-7fd7-4545-86a9-439ef63e92b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:54.236966 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.236907 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:49:54.282327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.277647 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:49:54.313344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313313 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.313539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313359 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.313539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313394 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.313539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313430 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.313539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313496 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.313539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313531 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.313862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313569 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.313862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313619 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.313862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313671 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.313862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313710 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.313862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313764 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.313862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313833 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.314172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313873 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.314172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.313924 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.314172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314133 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.314323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314193 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.314323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314267 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy\") pod \"e9fd3401-6d47-40fb-8333-db635f95f0ed\" (UID: \"e9fd3401-6d47-40fb-8333-db635f95f0ed\") " Nov 02 17:49:54.314323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314299 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token\") pod \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\" (UID: \"57dcf88a-e0b4-486e-8ec8-cbc04e075767\") " Nov 02 17:49:54.314847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314828 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.314928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314853 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dlw\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-kube-api-access-w6dlw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.314928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314867 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.314928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314883 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.314928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314897 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.314928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314911 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.314928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314925 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.315240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314939 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.315240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.314953 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/040bc6b3-7fd7-4545-86a9-439ef63e92b0-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.315240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.315046 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.315422 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.315403 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:54.316156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.315660 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.316156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.315947 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.316156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.316128 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.316447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.316294 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.317356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.317327 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.317501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.317443 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:49:54.317704 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.317681 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.318200 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.318158 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.322190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.322159 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:54.322309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.322257 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.323380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.323354 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:49:54.323468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.323407 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.323615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.323593 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.323674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.323623 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2" pod "57dcf88a-e0b4-486e-8ec8-cbc04e075767" (UID: "57dcf88a-e0b4-486e-8ec8-cbc04e075767"). InnerVolumeSpecName "kube-api-access-mzdk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.323974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.323952 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "kube-api-access-9kcz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:49:54.325023 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.325003 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy" pod "e9fd3401-6d47-40fb-8333-db635f95f0ed" (UID: "e9fd3401-6d47-40fb-8333-db635f95f0ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:49:54.401544 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.401510 5647 generic.go:356] "Generic (PLEG): container finished" podID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerID="76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8" exitCode=0 Nov 02 17:49:54.401721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.401648 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" event={"ID":"e9fd3401-6d47-40fb-8333-db635f95f0ed","Type":"ContainerDied","Data":"76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8"} Nov 02 17:49:54.401721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.401695 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" event={"ID":"e9fd3401-6d47-40fb-8333-db635f95f0ed","Type":"ContainerDied","Data":"4769a4b20ed7095afe2207ca1959cc555eda600b1ed5e5fce7f41eba42465af8"} Nov 02 17:49:54.401721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.401716 5647 scope.go:122] "RemoveContainer" containerID="76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8" Nov 02 17:49:54.401979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.401853 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl" Nov 02 17:49:54.405768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.405708 5647 generic.go:356] "Generic (PLEG): container finished" podID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerID="f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016" exitCode=0 Nov 02 17:49:54.405971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.405948 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" event={"ID":"040bc6b3-7fd7-4545-86a9-439ef63e92b0","Type":"ContainerDied","Data":"f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016"} Nov 02 17:49:54.406050 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.406004 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" event={"ID":"040bc6b3-7fd7-4545-86a9-439ef63e92b0","Type":"ContainerDied","Data":"e7630018caa1ab35a1b2f820e21b1f82f7ea1f1292b5157eb2f1a1bf3da81f3c"} Nov 02 17:49:54.406114 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.406093 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417031 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417070 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417088 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417103 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzdk2\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-kube-api-access-mzdk2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417119 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417132 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kcz5\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-kube-api-access-9kcz5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417148 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417161 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9fd3401-6d47-40fb-8333-db635f95f0ed-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417174 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417188 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417203 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417214 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417225 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417238 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417252 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9fd3401-6d47-40fb-8333-db635f95f0ed-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417266 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417279 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57dcf88a-e0b4-486e-8ec8-cbc04e075767-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.417293 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9fd3401-6d47-40fb-8333-db635f95f0ed-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.419377 5647 generic.go:356] "Generic (PLEG): container finished" podID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerID="e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e" exitCode=0 Nov 02 17:49:54.420095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.419492 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" event={"ID":"0349f5cd-00ad-4b9e-9be8-587f1ae34f60","Type":"ContainerDied","Data":"e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e"} Nov 02 17:49:54.421474 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.420906 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" Nov 02 17:49:54.421474 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.421441 5647 scope.go:122] "RemoveContainer" containerID="76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8" Nov 02 17:49:54.422941 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:54.422132 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8\": container with ID starting with 76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8 not found: ID does not exist" containerID="76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8" Nov 02 17:49:54.422941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.422169 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8"} err="failed to get container status \"76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8\": rpc error: code = NotFound desc = could not find container \"76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8\": container with ID starting with 76ca56af3ad16a032bf1ba92e0397571c841c4e5198c778060287c4e6b0400a8 not found: ID does not exist" Nov 02 17:49:54.422941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.422196 5647 scope.go:122] "RemoveContainer" containerID="f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016" Nov 02 17:49:54.422941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.422366 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb" event={"ID":"0349f5cd-00ad-4b9e-9be8-587f1ae34f60","Type":"ContainerDied","Data":"d999237c6a7f0f88584cbff9c80bc74e3c3ac63c18bdd62339a6275a5fa39b94"} Nov 02 17:49:54.425461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.425423 5647 generic.go:356] "Generic (PLEG): container finished" podID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerID="150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5" exitCode=0 Nov 02 17:49:54.425571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.425477 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" Nov 02 17:49:54.425571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.425503 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" event={"ID":"57dcf88a-e0b4-486e-8ec8-cbc04e075767","Type":"ContainerDied","Data":"150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5"} Nov 02 17:49:54.425696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.425589 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq" event={"ID":"57dcf88a-e0b4-486e-8ec8-cbc04e075767","Type":"ContainerDied","Data":"54b904eed89896ec979e995414f0c49dd02152bf63b561853b0e9cd1ccc41fb4"} Nov 02 17:49:54.443821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.443794 5647 scope.go:122] "RemoveContainer" containerID="19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3" Nov 02 17:49:54.454300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.454277 5647 scope.go:122] "RemoveContainer" containerID="f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016" Nov 02 17:49:54.454600 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:54.454577 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016\": container with ID starting with f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016 not found: ID does not exist" containerID="f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016" Nov 02 17:49:54.454693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.454611 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016"} err="failed to get container status \"f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016\": rpc error: code = NotFound desc = could not find container \"f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016\": container with ID starting with f68d6daacd383f115a44fcbb6f3abf389c0d8b26cf8e81591442d1375c230016 not found: ID does not exist" Nov 02 17:49:54.454693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.454634 5647 scope.go:122] "RemoveContainer" containerID="19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3" Nov 02 17:49:54.454902 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:54.454884 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3\": container with ID starting with 19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3 not found: ID does not exist" containerID="19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3" Nov 02 17:49:54.454974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.454911 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3"} err="failed to get container status \"19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3\": rpc error: code = NotFound desc = could not find container \"19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3\": container with ID starting with 19f734045090954c6e209fe95302af2cb04cfcc1d1a91910530cffcde7d547b3 not found: ID does not exist" Nov 02 17:49:54.454974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.454935 5647 scope.go:122] "RemoveContainer" containerID="e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e" Nov 02 17:49:54.465785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.465768 5647 scope.go:122] "RemoveContainer" containerID="e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e" Nov 02 17:49:54.466121 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:54.466098 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e\": container with ID starting with e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e not found: ID does not exist" containerID="e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e" Nov 02 17:49:54.466217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.466134 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e"} err="failed to get container status \"e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e\": rpc error: code = NotFound desc = could not find container \"e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e\": container with ID starting with e66bb8d4f73faeffcb998dbbea738f670193d8fa42d9d32bba551561aa1f3d8e not found: ID does not exist" Nov 02 17:49:54.466217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.466157 5647 scope.go:122] "RemoveContainer" containerID="150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5" Nov 02 17:49:54.476175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.476145 5647 scope.go:122] "RemoveContainer" containerID="150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5" Nov 02 17:49:54.476474 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:49:54.476452 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5\": container with ID starting with 150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5 not found: ID does not exist" containerID="150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5" Nov 02 17:49:54.476569 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.476479 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5"} err="failed to get container status \"150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5\": rpc error: code = NotFound desc = could not find container \"150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5\": container with ID starting with 150408480dc2d62e1b0514b2351584b4357ca2e42a147cae0d401ff648eb64f5 not found: ID does not exist" Nov 02 17:49:54.613409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.613378 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:49:54.613711 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.613691 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:49:54.614352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.614331 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:49:54.629507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.629481 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl"] Nov 02 17:49:54.642457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.637670 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-n5ngl"] Nov 02 17:49:54.818099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.818069 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq"] Nov 02 17:49:54.826762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.826703 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mv8sq"] Nov 02 17:49:54.939260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.939227 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb"] Nov 02 17:49:54.947600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:54.947572 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-7xkgb"] Nov 02 17:49:55.081829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.081722 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" path="/var/lib/kubelet/pods/0349f5cd-00ad-4b9e-9be8-587f1ae34f60/volumes" Nov 02 17:49:55.082285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.082271 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" path="/var/lib/kubelet/pods/57dcf88a-e0b4-486e-8ec8-cbc04e075767/volumes" Nov 02 17:49:55.082706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.082693 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" path="/var/lib/kubelet/pods/e9fd3401-6d47-40fb-8333-db635f95f0ed/volumes" Nov 02 17:49:55.083359 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.083344 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn"] Nov 02 17:49:55.090583 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.090542 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ad655-openshift-default-7896468c-mwdfn"] Nov 02 17:49:55.587432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.587402 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp"] Nov 02 17:49:55.587645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.587622 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" containerID="cri-o://40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e" gracePeriod=30 Nov 02 17:49:55.590708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.590683 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr"] Nov 02 17:49:55.591119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:55.591085 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" containerID="cri-o://36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3" gracePeriod=30 Nov 02 17:49:56.295956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:56.295905 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:49:57.095757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:57.093643 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" path="/var/lib/kubelet/pods/040bc6b3-7fd7-4545-86a9-439ef63e92b0/volumes" Nov 02 17:49:59.449352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:59.449305 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj"] Nov 02 17:49:59.449988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:49:59.449956 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" containerID="cri-o://975f8aaf4169d02924c42c8f09c286bfaa5c02bd2e7fe45ed2921f61af61cbc9" gracePeriod=30 Nov 02 17:50:00.948015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:00.947985 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:50:00.987659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:00.987634 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:50:01.089567 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089539 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.089761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089579 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.089761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089602 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.089761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.089761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089687 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.089761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089708 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.089928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089763 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.089928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089819 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.089928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089862 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.089928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089895 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.089928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089922 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.090081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089949 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.090081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.089997 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.090081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090041 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.090081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090062 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.090203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090080 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.090203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090133 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket\") pod \"116d27ce-62ce-42cc-a12d-386da00fef75\" (UID: \"116d27ce-62ce-42cc-a12d-386da00fef75\") " Nov 02 17:50:01.090203 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090175 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs\") pod \"13b8c1b5-e126-4171-aba8-6a6f970327d6\" (UID: \"13b8c1b5-e126-4171-aba8-6a6f970327d6\") " Nov 02 17:50:01.090601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090565 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.090816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090791 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.090913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090614 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:01.090913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090833 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.090913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090878 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.091101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.090923 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:01.091323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.091263 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.091479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.091456 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.091708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.091673 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.091708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.091698 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.093462 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.093425 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:01.093693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.093661 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:01.093899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.093812 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "kube-api-access-ksvht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:01.093899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.093874 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.094000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.093922 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:01.094100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.094082 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:01.094214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.094192 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy" pod "13b8c1b5-e126-4171-aba8-6a6f970327d6" (UID: "13b8c1b5-e126-4171-aba8-6a6f970327d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:01.094884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.094865 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp" pod "116d27ce-62ce-42cc-a12d-386da00fef75" (UID: "116d27ce-62ce-42cc-a12d-386da00fef75"). InnerVolumeSpecName "kube-api-access-22hjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193033 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193068 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193083 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d27ce-62ce-42cc-a12d-386da00fef75-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193097 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13b8c1b5-e126-4171-aba8-6a6f970327d6-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193111 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193126 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193139 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193153 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193166 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d27ce-62ce-42cc-a12d-386da00fef75-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193180 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193195 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvht\" (UniqueName: \"kubernetes.io/projected/13b8c1b5-e126-4171-aba8-6a6f970327d6-kube-api-access-ksvht\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193212 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193226 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22hjp\" (UniqueName: \"kubernetes.io/projected/116d27ce-62ce-42cc-a12d-386da00fef75-kube-api-access-22hjp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193241 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193255 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193269 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193282 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d27ce-62ce-42cc-a12d-386da00fef75-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.193380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.193297 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13b8c1b5-e126-4171-aba8-6a6f970327d6-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:01.533497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.533398 5647 generic.go:356] "Generic (PLEG): container finished" podID="116d27ce-62ce-42cc-a12d-386da00fef75" containerID="36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3" exitCode=0 Nov 02 17:50:01.533497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.533437 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" Nov 02 17:50:01.533497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.533461 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" event={"ID":"116d27ce-62ce-42cc-a12d-386da00fef75","Type":"ContainerDied","Data":"36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3"} Nov 02 17:50:01.533820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.533515 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr" event={"ID":"116d27ce-62ce-42cc-a12d-386da00fef75","Type":"ContainerDied","Data":"44638c6e30d76a40c29da51e8b7b47cf27b4c9bb169e0f520bb44cd5fc1346af"} Nov 02 17:50:01.533820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.533541 5647 scope.go:122] "RemoveContainer" containerID="36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3" Nov 02 17:50:01.535578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.535556 5647 generic.go:356] "Generic (PLEG): container finished" podID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerID="40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e" exitCode=0 Nov 02 17:50:01.535714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.535610 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" Nov 02 17:50:01.535794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.535753 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" event={"ID":"13b8c1b5-e126-4171-aba8-6a6f970327d6","Type":"ContainerDied","Data":"40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e"} Nov 02 17:50:01.535865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.535800 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp" event={"ID":"13b8c1b5-e126-4171-aba8-6a6f970327d6","Type":"ContainerDied","Data":"7f4dd8ba02f13bb67f6b377a8d8b9450025c2cb630d04f58124918ef0c2010eb"} Nov 02 17:50:01.545679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.545658 5647 scope.go:122] "RemoveContainer" containerID="36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3" Nov 02 17:50:01.545997 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:01.545975 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3\": container with ID starting with 36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3 not found: ID does not exist" containerID="36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3" Nov 02 17:50:01.546074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.546008 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3"} err="failed to get container status \"36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3\": rpc error: code = NotFound desc = could not find container \"36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3\": container with ID starting with 36eced31541dbb58144489c9d2dbd37cd00f0af9cc4d528d287b9914111a0be3 not found: ID does not exist" Nov 02 17:50:01.546074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.546026 5647 scope.go:122] "RemoveContainer" containerID="40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e" Nov 02 17:50:01.557159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.557138 5647 scope.go:122] "RemoveContainer" containerID="40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e" Nov 02 17:50:01.557497 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:01.557473 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e\": container with ID starting with 40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e not found: ID does not exist" containerID="40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e" Nov 02 17:50:01.557546 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.557510 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e"} err="failed to get container status \"40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e\": rpc error: code = NotFound desc = could not find container \"40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e\": container with ID starting with 40fb35e02f5644ff1a1a0def809c6b02d04a4ec9dedcf16c4171c50ea96b2d7e not found: ID does not exist" Nov 02 17:50:01.728521 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.728485 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp"] Nov 02 17:50:01.731387 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.731356 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-zc4bp"] Nov 02 17:50:01.844614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.844581 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr"] Nov 02 17:50:01.850062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:01.850034 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-w6rfr"] Nov 02 17:50:02.716045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.715998 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q"] Nov 02 17:50:02.717579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.717545 5647 cpu_manager.go:414] "containerMap: removing container" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" Nov 02 17:50:02.717777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.717760 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" Nov 02 17:50:02.717893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.717881 5647 cpu_manager.go:414] "containerMap: removing container" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" Nov 02 17:50:02.717975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.717964 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" Nov 02 17:50:02.718069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718059 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" Nov 02 17:50:02.718160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718149 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" Nov 02 17:50:02.718275 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718265 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" Nov 02 17:50:02.718350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718338 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" Nov 02 17:50:02.718434 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718424 5647 cpu_manager.go:414] "containerMap: removing container" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" Nov 02 17:50:02.718507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718497 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" Nov 02 17:50:02.718663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718651 5647 cpu_manager.go:414] "containerMap: removing container" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" Nov 02 17:50:02.722753 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.718768 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" Nov 02 17:50:02.723086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.723066 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" Nov 02 17:50:02.723205 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.723194 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="57dcf88a-e0b4-486e-8ec8-cbc04e075767" containerName="istio-proxy" Nov 02 17:50:02.723288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.723280 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9fd3401-6d47-40fb-8333-db635f95f0ed" containerName="istio-proxy" Nov 02 17:50:02.723369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.723358 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0349f5cd-00ad-4b9e-9be8-587f1ae34f60" containerName="istio-proxy" Nov 02 17:50:02.723436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.723428 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" containerName="istio-proxy" Nov 02 17:50:02.723528 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.723520 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" containerName="istio-proxy" Nov 02 17:50:02.738761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.738031 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.756614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.756573 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nh2\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.785948 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786011 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786103 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786147 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786230 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786309 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786390 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.789768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.786500 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.883696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.883606 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q"] Nov 02 17:50:02.892722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.892669 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.892949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.892783 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.892949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.892872 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.892949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.892912 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nh2\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.893148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.892979 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.893148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.893016 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.893148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.893085 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.893148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.893122 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.893372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.893187 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.931202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.905895 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.931202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.906208 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.931202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.906432 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.931202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.907053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:02.931202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.917154 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6f23bd-openshift-default-dockercfg-9wmkr\"" Nov 02 17:50:02.977862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:02.963973 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:03.020921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.020839 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:03.021620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.021590 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:03.022918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.022861 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:03.050770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.049973 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nh2\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:03.081962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.081928 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116d27ce-62ce-42cc-a12d-386da00fef75" path="/var/lib/kubelet/pods/116d27ce-62ce-42cc-a12d-386da00fef75/volumes" Nov 02 17:50:03.082514 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.082491 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13b8c1b5-e126-4171-aba8-6a6f970327d6" path="/var/lib/kubelet/pods/13b8c1b5-e126-4171-aba8-6a6f970327d6/volumes" Nov 02 17:50:03.298853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.298779 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:03.400316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.398591 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:50:03.400316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.399162 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:50:03.814784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.814719 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q"] Nov 02 17:50:03.851793 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:50:03.851728 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0753785_7233_4533_a398_0059cd0b8758.slice/crio-205cef2bfc1dc6268d298efd9e79a787ec2a08bf78f07d2c9d246a087dd26b74 WatchSource:0}: Error finding container 205cef2bfc1dc6268d298efd9e79a787ec2a08bf78f07d2c9d246a087dd26b74: Status 404 returned error can't find the container with id 205cef2bfc1dc6268d298efd9e79a787ec2a08bf78f07d2c9d246a087dd26b74 Nov 02 17:50:03.854386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.854354 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:03.854492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.854426 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:03.854492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:03.854455 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:04.599449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.599404 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" event={"ID":"a0753785-7233-4533-a398-0059cd0b8758","Type":"ContainerStarted","Data":"9aababaca92a01c536307e4c65145139419eac16cfe4ed272f9e8957311c9d13"} Nov 02 17:50:04.599689 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.599486 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" event={"ID":"a0753785-7233-4533-a398-0059cd0b8758","Type":"ContainerStarted","Data":"205cef2bfc1dc6268d298efd9e79a787ec2a08bf78f07d2c9d246a087dd26b74"} Nov 02 17:50:04.603600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.603569 5647 generic.go:356] "Generic (PLEG): container finished" podID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerID="975f8aaf4169d02924c42c8f09c286bfaa5c02bd2e7fe45ed2921f61af61cbc9" exitCode=0 Nov 02 17:50:04.603776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.603644 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" event={"ID":"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a","Type":"ContainerDied","Data":"975f8aaf4169d02924c42c8f09c286bfaa5c02bd2e7fe45ed2921f61af61cbc9"} Nov 02 17:50:04.603776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.603711 5647 scope.go:122] "RemoveContainer" containerID="034d94b5082456eb0b87b52ade49347a98c6a6b48b3fb515eaaf9f6557e23a4a" Nov 02 17:50:04.749035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.749007 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:50:04.826803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.826720 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podStartSLOduration=2.826705147 podStartE2EDuration="2.826705147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:04.777687304 +0000 UTC m=+1591.851214507" watchObservedRunningTime="2026-11-02 17:50:04.826705147 +0000 UTC m=+1591.900232350" Nov 02 17:50:04.848353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848315 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848419 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848454 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848518 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848486 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848640 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848683 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848710 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848796 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.848961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848910 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy\") pod \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\" (UID: \"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a\") " Nov 02 17:50:04.849018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.848968 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:04.849545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.849215 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:04.849545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.849311 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:50:04.849545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.849469 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:04.849545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.849486 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:04.850068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.850044 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.850195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.850072 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.850195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.850087 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.850195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.850104 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.850195 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.850118 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.851087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.851061 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "kube-api-access-hjdnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:04.851187 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.851153 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:50:04.851468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.851443 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:50:04.851709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.851680 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy" pod "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" (UID: "136f7b71-9fe6-41cf-9d10-bbd60ff34d1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:50:04.955759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.954888 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.955759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.954932 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjdnt\" (UniqueName: \"kubernetes.io/projected/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-kube-api-access-hjdnt\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.955759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.954947 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:04.955759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:04.954963 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:50:05.299407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.299375 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:05.304883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.304854 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:05.612389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.612358 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" Nov 02 17:50:05.612389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.612369 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj" event={"ID":"136f7b71-9fe6-41cf-9d10-bbd60ff34d1a","Type":"ContainerDied","Data":"672d28f4f0c297791e2f71c2d3e46ad2592da29217937beba32bdc8b6a4e18ee"} Nov 02 17:50:05.612584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.612444 5647 scope.go:122] "RemoveContainer" containerID="975f8aaf4169d02924c42c8f09c286bfaa5c02bd2e7fe45ed2921f61af61cbc9" Nov 02 17:50:05.613516 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.613495 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:05.614157 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.614134 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:50:05.919450 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.919366 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj"] Nov 02 17:50:05.923658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:05.923628 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-494cff-openshift-default-84599fbf5-gt9sj"] Nov 02 17:50:05.948666 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:05.948622 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:50:07.082047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.082005 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" path="/var/lib/kubelet/pods/136f7b71-9fe6-41cf-9d10-bbd60ff34d1a/volumes" Nov 02 17:50:07.629719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.629677 5647 generic.go:356] "Generic (PLEG): container finished" podID="752613f5-4c69-4c73-a417-93325548fc69" containerID="b3faf3307456bd8a2c1c578de252d61e01d452e45d55ddd16b256f80d1b688af" exitCode=0 Nov 02 17:50:07.629923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.629776 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" event={"ID":"752613f5-4c69-4c73-a417-93325548fc69","Type":"ContainerDied","Data":"b3faf3307456bd8a2c1c578de252d61e01d452e45d55ddd16b256f80d1b688af"} Nov 02 17:50:07.630402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.630382 5647 scope.go:122] "RemoveContainer" containerID="b3faf3307456bd8a2c1c578de252d61e01d452e45d55ddd16b256f80d1b688af" Nov 02 17:50:07.631888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.631850 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:07.632008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.631927 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:07.632008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:07.631973 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:08.639515 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:08.639476 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" event={"ID":"752613f5-4c69-4c73-a417-93325548fc69","Type":"ContainerStarted","Data":"1a8f028d1283821f1a23288491dc5bf1f4180a93026f6721ad42189321a82f1c"} Nov 02 17:50:08.732051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:08.732015 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:50:08.733297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:08.733267 5647 patch_prober.go:29] interesting pod/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:50:08.733431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:08.733317 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 17:50:09.732408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:09.732369 5647 patch_prober.go:29] interesting pod/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:50:09.732845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:09.732422 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 17:50:09.835508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:09.835471 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:50:10.732573 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.732537 5647 patch_prober.go:29] interesting pod/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:50:10.733164 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.732607 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 17:50:10.893649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.893606 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9"] Nov 02 17:50:10.894521 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894499 5647 cpu_manager.go:414] "containerMap: removing container" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" Nov 02 17:50:10.894577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894527 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" Nov 02 17:50:10.894613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894589 5647 cpu_manager.go:414] "containerMap: removing container" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" Nov 02 17:50:10.894613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894599 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" Nov 02 17:50:10.894749 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894722 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" Nov 02 17:50:10.894790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894759 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" Nov 02 17:50:10.894790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.894775 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="040bc6b3-7fd7-4545-86a9-439ef63e92b0" containerName="istio-proxy" Nov 02 17:50:10.899041 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.899016 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:10.962611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:10.962576 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9"] Nov 02 17:50:11.046570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.046480 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8cbedb9f-serving-cert\"" Nov 02 17:50:11.056437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.056400 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.056596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.056491 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7z7\" (UniqueName: \"kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.056651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.056625 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.056718 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.056701 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.072000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.071968 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 17:50:11.072000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.071983 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94n2w\"" Nov 02 17:50:11.157702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.157660 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.157901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.157789 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7z7\" (UniqueName: \"kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.157901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.157853 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.157901 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:11.157856 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-serving-cert: secret "maas-api-e2e-mt-8cbedb9f-serving-cert" not found Nov 02 17:50:11.158078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.157909 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.158078 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:11.157918 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls podName:047c0850-a124-4e4e-ac98-380c15f7e971 nodeName:}" failed. No retries permitted until 2026-11-02 17:50:11.657901828 +0000 UTC m=+1598.731429046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls") pod "maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" (UID: "047c0850-a124-4e4e-ac98-380c15f7e971") : secret "maas-api-e2e-mt-8cbedb9f-serving-cert" not found Nov 02 17:50:11.158628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.158600 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.160487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.160468 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.230566 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.230535 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7z7\" (UniqueName: \"kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.557286 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.557242 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz"] Nov 02 17:50:11.558123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.558103 5647 cpu_manager.go:414] "containerMap: removing container" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" Nov 02 17:50:11.558194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.558125 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136f7b71-9fe6-41cf-9d10-bbd60ff34d1a" containerName="istio-proxy" Nov 02 17:50:11.562014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.561995 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.622048 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.622013 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6f23bd-dockercfg-rk5zv\"" Nov 02 17:50:11.622589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.622566 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6f23bd\"" Nov 02 17:50:11.649763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.645698 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz"] Nov 02 17:50:11.670806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.668690 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.670806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.668906 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcb2\" (UniqueName: \"kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2\") pod \"payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.670806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.668948 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.751769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.736118 5647 patch_prober.go:29] interesting pod/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:50:11.751769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.736183 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 17:50:11.755519 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.755452 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\") pod \"maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.791044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.783076 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcb2\" (UniqueName: \"kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2\") pod \"payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.791044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.783143 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.791044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.783793 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr"] Nov 02 17:50:11.791044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.784407 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.791044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.789399 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:11.863624 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.863587 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr"] Nov 02 17:50:11.872785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.872715 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcb2\" (UniqueName: \"kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2\") pod \"payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.874212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.874187 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:11.882850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.881500 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:11.885926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.885901 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:11.886176 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.886158 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpr9\" (UniqueName: \"kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.013039 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.990073 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.013039 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.990189 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpr9\" (UniqueName: \"kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.013039 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:11.991168 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.049703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.049666 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpr9\" (UniqueName: \"kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.145788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.145726 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.354919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.354883 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9"] Nov 02 17:50:12.356135 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:50:12.356108 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047c0850_a124_4e4e_ac98_380c15f7e971.slice/crio-8a3514a1b9cace33b71efea41fb311bf73efabde5f9bc8b5303449bc6942f396 WatchSource:0}: Error finding container 8a3514a1b9cace33b71efea41fb311bf73efabde5f9bc8b5303449bc6942f396: Status 404 returned error can't find the container with id 8a3514a1b9cace33b71efea41fb311bf73efabde5f9bc8b5303449bc6942f396 Nov 02 17:50:12.359647 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.359623 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz"] Nov 02 17:50:12.391929 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:50:12.391894 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884cc241_6b33_4240_8f51_3ac590b01b39.slice/crio-dde22b81b9c2424ba2dfc724738783d31d24d210ef19e739013e339e9eceedb1 WatchSource:0}: Error finding container dde22b81b9c2424ba2dfc724738783d31d24d210ef19e739013e339e9eceedb1: Status 404 returned error can't find the container with id dde22b81b9c2424ba2dfc724738783d31d24d210ef19e739013e339e9eceedb1 Nov 02 17:50:12.563126 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:50:12.563090 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2112269_cd3d_4659_8156_9380e128dbde.slice/crio-f48995a6d947d8a93ee6775dc899b83daa4dff2a6b9e0ba0f0bb57dcd0497df6 WatchSource:0}: Error finding container f48995a6d947d8a93ee6775dc899b83daa4dff2a6b9e0ba0f0bb57dcd0497df6: Status 404 returned error can't find the container with id f48995a6d947d8a93ee6775dc899b83daa4dff2a6b9e0ba0f0bb57dcd0497df6 Nov 02 17:50:12.564099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.564077 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr"] Nov 02 17:50:12.732839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.732786 5647 patch_prober.go:29] interesting pod/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:50:12.732999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.732857 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 17:50:12.841004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.840948 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" event={"ID":"884cc241-6b33-4240-8f51-3ac590b01b39","Type":"ContainerStarted","Data":"00396f18f98495a1bdd62731fc02c37c507898f5062924c17fb46f5b9fafaa89"} Nov 02 17:50:12.841410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.841099 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" event={"ID":"884cc241-6b33-4240-8f51-3ac590b01b39","Type":"ContainerStarted","Data":"dde22b81b9c2424ba2dfc724738783d31d24d210ef19e739013e339e9eceedb1"} Nov 02 17:50:12.841410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.841131 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:12.843220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.843186 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" event={"ID":"d2112269-cd3d-4659-8156-9380e128dbde","Type":"ContainerStarted","Data":"066fc7d9a2322349a3f67faa3e6c993b2f1b8a44c9f7df4f37f146e650fcc8d3"} Nov 02 17:50:12.843372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.843241 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" event={"ID":"d2112269-cd3d-4659-8156-9380e128dbde","Type":"ContainerStarted","Data":"f48995a6d947d8a93ee6775dc899b83daa4dff2a6b9e0ba0f0bb57dcd0497df6"} Nov 02 17:50:12.843372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.843296 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:12.844537 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.844515 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" event={"ID":"047c0850-a124-4e4e-ac98-380c15f7e971","Type":"ContainerStarted","Data":"8a3514a1b9cace33b71efea41fb311bf73efabde5f9bc8b5303449bc6942f396"} Nov 02 17:50:12.962085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:12.961979 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" podStartSLOduration=1.961965074 podStartE2EDuration="1.961965074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:12.959197527 +0000 UTC m=+1600.032724731" watchObservedRunningTime="2026-11-02 17:50:12.961965074 +0000 UTC m=+1600.035492277" Nov 02 17:50:13.148085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:13.148018 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" podStartSLOduration=2.147997707 podStartE2EDuration="2.147997707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:13.088309791 +0000 UTC m=+1600.161837001" watchObservedRunningTime="2026-11-02 17:50:13.147997707 +0000 UTC m=+1600.221524907" Nov 02 17:50:13.732554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:13.732516 5647 patch_prober.go:29] interesting pod/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 02 17:50:13.732801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:13.732600 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 02 17:50:14.737320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:14.737290 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:50:14.738344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:14.738323 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:50:14.864097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:14.864056 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" event={"ID":"047c0850-a124-4e4e-ac98-380c15f7e971","Type":"ContainerStarted","Data":"6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2"} Nov 02 17:50:14.864276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:14.864161 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:15.018975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:15.018850 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" podStartSLOduration=2.72379117 podStartE2EDuration="5.018836253s" totalImagesPullingTime="2.295045083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:15.016130627 +0000 UTC m=+1602.089657831" watchObservedRunningTime="2026-11-02 17:50:15.018836253 +0000 UTC m=+1602.092363455" Nov 02 17:50:20.877337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:20.877307 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:50:23.858247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:23.858217 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:50:23.858636 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:23.858292 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:50:26.077860 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:26.077821 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:50:35.231755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.208880 5647 scope.go:122] "RemoveContainer" containerID="58f008e8111764ac8ac55d36b5342fb1a1af82faec4ae705f4107342b9ab7829" Nov 02 17:50:35.301033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.301005 5647 scope.go:122] "RemoveContainer" containerID="35b150800f5295ad85816f541305f28c4cfce4de84175dc2d6d001ecdddbc79c" Nov 02 17:50:35.335799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.335769 5647 scope.go:122] "RemoveContainer" containerID="319f150a779682161d9090b46d0e27b1e0d077bbf8904f4ca0012e8dfe291dec" Nov 02 17:50:35.345291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.345269 5647 scope.go:122] "RemoveContainer" containerID="dc5ffaa70a9ebb9ba84ebf047504a5190d08bad24a0fc55f57abb7b1026a8da1" Nov 02 17:50:35.354702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.354681 5647 scope.go:122] "RemoveContainer" containerID="9240734f47463a39627edf65e2adf16fa760bbf8cc0b47214d82bcda61d64c26" Nov 02 17:50:35.363912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.363890 5647 scope.go:122] "RemoveContainer" containerID="9363eb0db204d831960fa8f58288f1a43bd5f2b6afd2a511ce6bc18ee6383bf6" Nov 02 17:50:35.373167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.373142 5647 scope.go:122] "RemoveContainer" containerID="0f69a99ec5475e0a6416170ee8629f3cec01ed2221cb870c83cf8fc3c26a3206" Nov 02 17:50:35.381996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:35.381975 5647 scope.go:122] "RemoveContainer" containerID="033402b0431dda235dde9dd2ff8575209a3cce09e9b677850cd428acdf880a50" Nov 02 17:50:35.956261 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:35.956220 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:50:43.092631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:43.092596 5647 generic.go:356] "Generic (PLEG): container finished" podID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerID="7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f" exitCode=0 Nov 02 17:50:43.093097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:43.092681 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" event={"ID":"3f284f18-defa-49a4-8b36-52ff0ef8eb13","Type":"ContainerDied","Data":"7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f"} Nov 02 17:50:43.093143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:43.093125 5647 scope.go:122] "RemoveContainer" containerID="7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f" Nov 02 17:50:43.094782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:43.094722 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:43.094874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:43.094821 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:43.094874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:43.094853 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:44.114763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:44.101966 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" event={"ID":"3f284f18-defa-49a4-8b36-52ff0ef8eb13","Type":"ContainerStarted","Data":"873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741"} Nov 02 17:50:44.520182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:44.520091 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:50:44.525773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:44.524905 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:50:44.525773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:44.524989 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 17:50:44.922760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:44.919856 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:50:45.527177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:45.527122 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:50:45.527617 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:45.527211 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 17:50:46.528136 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:46.528097 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:50:46.528595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:46.528181 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 17:50:47.528892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:47.528853 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:50:47.529519 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:47.529486 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 17:50:48.526763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:48.522268 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 02 17:50:48.526763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:48.522350 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 02 17:50:49.522820 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.522784 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:50:49.523724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.523698 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:50:49.633791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.633755 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l"] Nov 02 17:50:49.639950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.639928 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.671446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.671416 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l"] Nov 02 17:50:49.693674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.693646 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-27ab38-serving-cert\"" Nov 02 17:50:49.705258 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.705220 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.705418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.705384 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.705466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.705416 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vpf\" (UniqueName: \"kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.705527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.705513 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.807119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.807013 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.807119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.807071 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vpf\" (UniqueName: \"kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.807375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.807137 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.807375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.807235 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.807375 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:49.807306 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-serving-cert: secret "maas-api-e2e-api-b-27ab38-serving-cert" not found Nov 02 17:50:49.807375 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:50:49.807367 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls podName:cb758a75-cf10-4257-a29e-c3f05157d38d nodeName:}" failed. No retries permitted until 2026-11-02 17:50:50.307351183 +0000 UTC m=+1637.380878364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls") pod "maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" (UID: "cb758a75-cf10-4257-a29e-c3f05157d38d") : secret "maas-api-e2e-api-b-27ab38-serving-cert" not found Nov 02 17:50:49.807772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.807753 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.809894 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.809875 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:49.869511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:49.869478 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vpf\" (UniqueName: \"kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:50.320005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:50.319952 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:50.322620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:50.322593 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\") pod \"maas-api-e2e-api-b-27ab38-848dcc546d-zn48l\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:50.554280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:50.554244 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:50.949060 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:50:50.949028 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb758a75_cf10_4257_a29e_c3f05157d38d.slice/crio-9a42142fdb513f4803995721d989e1942add76f65ade591e5a0b3d87c69b369e WatchSource:0}: Error finding container 9a42142fdb513f4803995721d989e1942add76f65ade591e5a0b3d87c69b369e: Status 404 returned error can't find the container with id 9a42142fdb513f4803995721d989e1942add76f65ade591e5a0b3d87c69b369e Nov 02 17:50:50.953165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:50.953138 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l"] Nov 02 17:50:51.196811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:51.196702 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" event={"ID":"cb758a75-cf10-4257-a29e-c3f05157d38d","Type":"ContainerStarted","Data":"9a42142fdb513f4803995721d989e1942add76f65ade591e5a0b3d87c69b369e"} Nov 02 17:50:52.205256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:52.205215 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" event={"ID":"cb758a75-cf10-4257-a29e-c3f05157d38d","Type":"ContainerStarted","Data":"9abc5820d83119ec202f6e0b4c48d2d4506c5b2653ed6b4422441a45f61beaa5"} Nov 02 17:50:52.205791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:52.205344 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:52.328217 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:52.328159 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" podStartSLOduration=2.554157027 podStartE2EDuration="3.328145258s" totalImagesPullingTime="773.988231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:52.326612756 +0000 UTC m=+1639.400139961" watchObservedRunningTime="2026-11-02 17:50:52.328145258 +0000 UTC m=+1639.401672461" Nov 02 17:50:55.609889 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.609856 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd"] Nov 02 17:50:55.614262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.614241 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711292 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711344 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711370 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711608 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711645 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711661 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711800 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvs4\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711850 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.711992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.711960 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813180 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813140 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813194 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813255 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813395 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813460 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813597 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813616 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813655 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813786 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813685 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813786 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813714 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813906 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813803 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvs4\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.813906 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.813850 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.814087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.814060 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.814444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.814421 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.816396 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.816375 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.816869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.816832 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.824204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.824172 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd"] Nov 02 17:50:55.878307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.878238 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.878307 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.878302 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvs4\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:55.928867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:55.928836 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:56.369908 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:56.369876 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd"] Nov 02 17:50:56.370720 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:50:56.370696 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf070449e_b50d_405b_abeb_b251576ed8de.slice/crio-bdfaf0a5a0f65849aa2090e4cc436ff13728386c8089cef1da4216c9c836354a WatchSource:0}: Error finding container bdfaf0a5a0f65849aa2090e4cc436ff13728386c8089cef1da4216c9c836354a: Status 404 returned error can't find the container with id bdfaf0a5a0f65849aa2090e4cc436ff13728386c8089cef1da4216c9c836354a Nov 02 17:50:56.373323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:56.373284 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:56.373443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:56.373372 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:56.373443 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:56.373416 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:50:57.239420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:57.239382 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" event={"ID":"f070449e-b50d-405b-abeb-b251576ed8de","Type":"ContainerStarted","Data":"39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17"} Nov 02 17:50:57.239920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:57.239442 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" event={"ID":"f070449e-b50d-405b-abeb-b251576ed8de","Type":"ContainerStarted","Data":"bdfaf0a5a0f65849aa2090e4cc436ff13728386c8089cef1da4216c9c836354a"} Nov 02 17:50:57.432410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:57.432335 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podStartSLOduration=2.432307154 podStartE2EDuration="2.432307154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:57.426273654 +0000 UTC m=+1644.499800861" watchObservedRunningTime="2026-11-02 17:50:57.432307154 +0000 UTC m=+1644.505834359" Nov 02 17:50:57.929932 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:57.929893 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:50:57.931642 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:57.931614 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 17:50:57.931821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:57.931669 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 17:50:58.232461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:58.232383 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:50:58.930765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:58.930358 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 17:50:58.930765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:58.930451 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 17:50:59.930056 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:59.930022 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 17:50:59.930243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:50:59.930074 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 17:51:00.929949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:00.929913 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 17:51:00.930377 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:00.929969 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 17:51:01.930227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:01.930182 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Nov 02 17:51:01.930664 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:01.930241 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Nov 02 17:51:03.044146 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:03.044104 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:51:03.045100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:03.045061 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:51:03.046314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:03.046290 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:51:04.059654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:04.059612 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl"] Nov 02 17:51:04.110198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:04.110167 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6159bd21-kl7rl"] Nov 02 17:51:05.093175 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.093139 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e4f831-1522-4092-bcf9-514e9a243b16" path="/var/lib/kubelet/pods/49e4f831-1522-4092-bcf9-514e9a243b16/volumes" Nov 02 17:51:05.787636 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.787601 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx"] Nov 02 17:51:05.792958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.792933 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:05.849934 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.849903 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6f23bd-serving-cert\"" Nov 02 17:51:05.904069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.904025 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx"] Nov 02 17:51:05.907436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.907410 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:05.907595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.907502 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:05.907691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.907665 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n776c\" (UniqueName: \"kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:05.907786 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:05.907765 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:05.953086 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:05.953046 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:51:06.008726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.008694 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.008953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.008826 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n776c\" (UniqueName: \"kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.008953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.008864 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.008953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.008948 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.009445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.009406 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.011426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.011407 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.011525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.011449 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.069916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.069843 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n776c\" (UniqueName: \"kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c\") pod \"maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.107492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.107450 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:06.496992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:06.496966 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx"] Nov 02 17:51:06.497472 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:06.497446 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ea30ba_2a57_4111_ab0e_c8b9eb458785.slice/crio-0360e6ec668a0a8b5eaa296c669fd02a06d647cd85c2d46bb1be4596e5978486 WatchSource:0}: Error finding container 0360e6ec668a0a8b5eaa296c669fd02a06d647cd85c2d46bb1be4596e5978486: Status 404 returned error can't find the container with id 0360e6ec668a0a8b5eaa296c669fd02a06d647cd85c2d46bb1be4596e5978486 Nov 02 17:51:07.386542 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:07.386511 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" event={"ID":"e7ea30ba-2a57-4111-ab0e-c8b9eb458785","Type":"ContainerStarted","Data":"0360e6ec668a0a8b5eaa296c669fd02a06d647cd85c2d46bb1be4596e5978486"} Nov 02 17:51:07.989561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:07.989524 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx"] Nov 02 17:51:07.989778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:07.989725 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" containerID="cri-o://ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb" gracePeriod=30 Nov 02 17:51:08.394647 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:08.394609 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" event={"ID":"e7ea30ba-2a57-4111-ab0e-c8b9eb458785","Type":"ContainerStarted","Data":"dd271de8bb6f7fe9a704954aa033e890f86b3b1217bab946605a412715e45080"} Nov 02 17:51:08.395096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:08.394685 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:08.541482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:08.541432 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" podStartSLOduration=2.7435675379999998 podStartE2EDuration="3.541416618s" totalImagesPullingTime="797.84908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:08.539975803 +0000 UTC m=+1655.613503007" watchObservedRunningTime="2026-11-02 17:51:08.541416618 +0000 UTC m=+1655.614943821" Nov 02 17:51:10.821641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.821604 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv"] Nov 02 17:51:10.825864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.825844 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900387 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900350 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900407 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900503 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900558 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900645 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900689 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvnh\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900782 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900850 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.900996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.900973 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:10.917079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.917047 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:51:10.951003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:10.950972 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv"] Nov 02 17:51:11.002127 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002087 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002141 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002185 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002240 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002267 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002292 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvnh\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002348 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002429 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002556 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002669 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.002810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.002726 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.003102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.003016 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.003102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.003091 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.003102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.003108 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.004928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.004907 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.005027 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.005000 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.067005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.066967 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvnh\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.069148 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.069122 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.139697 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.139658 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:11.180262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.180223 5647 patch_prober.go:29] interesting pod/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:51:11.180418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.180292 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:51:11.580271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.580236 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv"] Nov 02 17:51:11.582209 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:11.582178 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3375acde_3ce8_442a_9718_e7a6799e9722.slice/crio-4a6fa4d1937a994d11ec771e7d1d82eac03255512a8d6f733b8fc599048265f1 WatchSource:0}: Error finding container 4a6fa4d1937a994d11ec771e7d1d82eac03255512a8d6f733b8fc599048265f1: Status 404 returned error can't find the container with id 4a6fa4d1937a994d11ec771e7d1d82eac03255512a8d6f733b8fc599048265f1 Nov 02 17:51:11.585219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.585174 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:11.585340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.585261 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:11.585340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:11.585314 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:12.430265 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:12.430221 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" event={"ID":"3375acde-3ce8-442a-9718-e7a6799e9722","Type":"ContainerStarted","Data":"fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b"} Nov 02 17:51:12.430849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:12.430822 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" event={"ID":"3375acde-3ce8-442a-9718-e7a6799e9722","Type":"ContainerStarted","Data":"4a6fa4d1937a994d11ec771e7d1d82eac03255512a8d6f733b8fc599048265f1"} Nov 02 17:51:12.566846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:12.566792 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podStartSLOduration=2.566775199 podStartE2EDuration="2.566775199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:12.565096128 +0000 UTC m=+1659.638623334" watchObservedRunningTime="2026-11-02 17:51:12.566775199 +0000 UTC m=+1659.640302406" Nov 02 17:51:13.156491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.156461 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:13.158560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.158341 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:51:13.158560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.158387 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 17:51:13.472915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.472886 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:51:13.473296 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.473262 5647 generic.go:356] "Generic (PLEG): container finished" podID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerID="ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb" exitCode=0 Nov 02 17:51:13.474021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.473999 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" event={"ID":"e6a491df-b1ac-4bd6-9a99-e114a01c35dc","Type":"ContainerDied","Data":"ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb"} Nov 02 17:51:13.474216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.474202 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" event={"ID":"e6a491df-b1ac-4bd6-9a99-e114a01c35dc","Type":"ContainerDied","Data":"96265f067301b5829932392080e584479477d9cfa7be57f0468192419d94cdca"} Nov 02 17:51:13.474337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.474224 5647 scope.go:122] "RemoveContainer" containerID="ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb" Nov 02 17:51:13.485541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.485519 5647 scope.go:122] "RemoveContainer" containerID="201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2" Nov 02 17:51:13.495866 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.495844 5647 scope.go:122] "RemoveContainer" containerID="ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb" Nov 02 17:51:13.496179 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:13.496159 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb\": container with ID starting with ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb not found: ID does not exist" containerID="ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb" Nov 02 17:51:13.496234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.496189 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb"} err="failed to get container status \"ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb\": rpc error: code = NotFound desc = could not find container \"ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb\": container with ID starting with ca6275c7c6e1c1e88aa2e404a6aecf77c21336a50c0e1cde25ec56488b634dfb not found: ID does not exist" Nov 02 17:51:13.496234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.496207 5647 scope.go:122] "RemoveContainer" containerID="201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2" Nov 02 17:51:13.496510 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:13.496465 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2\": container with ID starting with 201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2 not found: ID does not exist" containerID="201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2" Nov 02 17:51:13.496510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.496491 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2"} err="failed to get container status \"201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2\": rpc error: code = NotFound desc = could not find container \"201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2\": container with ID starting with 201dd2643d70bcc705e8a4572b4289f80e7529972b01a9953b8dedb41df93cf2 not found: ID does not exist" Nov 02 17:51:13.566961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.566866 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.566961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.566948 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567031 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567056 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567139 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:13.567222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567185 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567256 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567306 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567339 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567385 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert\") pod \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\" (UID: \"e6a491df-b1ac-4bd6-9a99-e114a01c35dc\") " Nov 02 17:51:13.567758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567708 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:13.567849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567800 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:13.567941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.567917 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:13.568652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.568625 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:13.568791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.568765 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.568907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.568886 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.568977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.568916 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.568977 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.568933 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.570759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.570633 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "kube-api-access-z797n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:13.570759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.570671 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:13.570759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.570694 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:13.571043 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.571017 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token" pod "e6a491df-b1ac-4bd6-9a99-e114a01c35dc" (UID: "e6a491df-b1ac-4bd6-9a99-e114a01c35dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:13.671199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.670830 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z797n\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-kube-api-access-z797n\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.671199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.670869 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.671199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.670884 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.671199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.670898 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:13.671199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:13.670911 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6a491df-b1ac-4bd6-9a99-e114a01c35dc-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:14.142186 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:14.142147 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:51:14.142529 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:14.142491 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 17:51:14.429818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:14.428916 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:51:14.481902 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:14.481866 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx" Nov 02 17:51:14.761855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:14.761569 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx"] Nov 02 17:51:14.765683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:14.765659 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-215cbd-openshift-default-7449444b59-77cnx"] Nov 02 17:51:15.084119 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:15.084029 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" path="/var/lib/kubelet/pods/e6a491df-b1ac-4bd6-9a99-e114a01c35dc/volumes" Nov 02 17:51:15.143763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:15.142934 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:51:15.143763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:15.142996 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 17:51:16.141763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:16.141174 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:51:16.141763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:16.141240 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 17:51:17.213761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:17.172451 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:51:17.221353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:17.221304 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 17:51:18.144183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:18.144145 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Nov 02 17:51:18.144498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:18.144470 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Nov 02 17:51:19.145548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:19.145515 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:19.145974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:19.145950 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:19.146768 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:19.146728 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:51:22.555411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:22.555321 5647 generic.go:356] "Generic (PLEG): container finished" podID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerID="b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca" exitCode=0 Nov 02 17:51:22.555411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:22.555385 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" event={"ID":"5171dfce-2108-4f03-8b01-c0fd91b3e5a3","Type":"ContainerDied","Data":"b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca"} Nov 02 17:51:22.555892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:22.555828 5647 scope.go:122] "RemoveContainer" containerID="b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca" Nov 02 17:51:22.557478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:22.557443 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:22.557592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:22.557513 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:22.557592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:22.557547 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:23.703783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:23.700364 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" event={"ID":"5171dfce-2108-4f03-8b01-c0fd91b3e5a3","Type":"ContainerStarted","Data":"a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae"} Nov 02 17:51:24.659006 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:24.658959 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:51:24.659218 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:24.659160 5647 patch_prober.go:29] interesting pod/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 17:51:24.659295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:24.659221 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 17:51:25.658535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.658503 5647 patch_prober.go:29] interesting pod/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 17:51:25.658938 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.658572 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 17:51:25.928367 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.928228 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv"] Nov 02 17:51:25.929274 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.929222 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" Nov 02 17:51:25.929424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.929276 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" Nov 02 17:51:25.929424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.929312 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" Nov 02 17:51:25.929424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.929320 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" Nov 02 17:51:25.929550 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.929509 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" Nov 02 17:51:25.933692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:25.933670 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.036146 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.036108 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv"] Nov 02 17:51:26.080588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.080557 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.080811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.080601 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.080811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.080623 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.080811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.080710 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.080811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.080774 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cn9k\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.081051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.080967 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.081051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.081016 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.081129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.081077 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.081173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.081134 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182436 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182511 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182541 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182575 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182625 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cn9k\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182839 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182880 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182904 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182972 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.182997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.182984 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.183248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.183056 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.183289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.183270 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.183345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.183322 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.183655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.183637 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.185010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.184992 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.185580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.185560 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.242077 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.242048 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.242239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.242143 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cn9k\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k\") pod \"e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.249211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.248978 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:26.659127 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.659078 5647 patch_prober.go:29] interesting pod/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 17:51:26.659535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.659137 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 17:51:26.751433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.751402 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv"] Nov 02 17:51:26.751606 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:26.751585 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d228d91_771d_42af_8750_ce9b823760ed.slice/crio-40c56cccb067a0de559ee337d3faa1c9740581cda3020b0107154459d10a90e8 WatchSource:0}: Error finding container 40c56cccb067a0de559ee337d3faa1c9740581cda3020b0107154459d10a90e8: Status 404 returned error can't find the container with id 40c56cccb067a0de559ee337d3faa1c9740581cda3020b0107154459d10a90e8 Nov 02 17:51:26.754159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.754127 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:26.754237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.754191 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:26.754237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:26.754222 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:27.659287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:27.659252 5647 patch_prober.go:29] interesting pod/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 17:51:27.659723 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:27.659306 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 17:51:27.760316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:27.760282 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" event={"ID":"2d228d91-771d-42af-8750-ce9b823760ed","Type":"ContainerStarted","Data":"07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285"} Nov 02 17:51:27.760495 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:27.760335 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" event={"ID":"2d228d91-771d-42af-8750-ce9b823760ed","Type":"ContainerStarted","Data":"40c56cccb067a0de559ee337d3faa1c9740581cda3020b0107154459d10a90e8"} Nov 02 17:51:27.945824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:27.941814 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podStartSLOduration=2.941787854 podStartE2EDuration="2.941787854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:27.929278197 +0000 UTC m=+1675.002805403" watchObservedRunningTime="2026-11-02 17:51:27.941787854 +0000 UTC m=+1675.015315048" Nov 02 17:51:28.255219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:28.255114 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:28.256808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:28.256780 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:51:28.256947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:28.256829 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 17:51:28.658452 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:28.658407 5647 patch_prober.go:29] interesting pod/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Nov 02 17:51:28.658674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:28.658472 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Nov 02 17:51:29.255313 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.255273 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:51:29.255781 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.255339 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 17:51:29.599113 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.599074 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7529-21c899"] Nov 02 17:51:29.599862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.599846 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a491df-b1ac-4bd6-9a99-e114a01c35dc" containerName="istio-proxy" Nov 02 17:51:29.623289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.623255 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:29.656351 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.656321 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-21c899"] Nov 02 17:51:29.664480 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.664438 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:51:29.664874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.664850 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:51:29.666903 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.666879 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:51:29.677151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.677026 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7kk\" (UniqueName: \"kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk\") pod \"test-curl-7529-21c899\" (UID: \"e3634fd1-cce3-4d09-9570-c1a16b10c24d\") " pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:29.724762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.712997 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4gwc\"" Nov 02 17:51:29.778873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.778842 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7kk\" (UniqueName: \"kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk\") pod \"test-curl-7529-21c899\" (UID: \"e3634fd1-cce3-4d09-9570-c1a16b10c24d\") " pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:29.838666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.838624 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7kk\" (UniqueName: \"kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk\") pod \"test-curl-7529-21c899\" (UID: \"e3634fd1-cce3-4d09-9570-c1a16b10c24d\") " pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:29.940224 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:29.940126 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:30.267128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:30.264258 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:51:30.267128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:30.264316 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 17:51:30.391347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:30.391319 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-21c899"] Nov 02 17:51:30.392539 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:30.392508 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3634fd1_cce3_4d09_9570_c1a16b10c24d.slice/crio-07c0b8e2ce4642e0bca2a8f931e866dd0ca919cdce3bad472483faf4ecd2a09b WatchSource:0}: Error finding container 07c0b8e2ce4642e0bca2a8f931e866dd0ca919cdce3bad472483faf4ecd2a09b: Status 404 returned error can't find the container with id 07c0b8e2ce4642e0bca2a8f931e866dd0ca919cdce3bad472483faf4ecd2a09b Nov 02 17:51:30.783048 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:30.783007 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-21c899" event={"ID":"e3634fd1-cce3-4d09-9570-c1a16b10c24d","Type":"ContainerStarted","Data":"f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee"} Nov 02 17:51:30.783245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:30.783056 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-21c899" event={"ID":"e3634fd1-cce3-4d09-9570-c1a16b10c24d","Type":"ContainerStarted","Data":"07c0b8e2ce4642e0bca2a8f931e866dd0ca919cdce3bad472483faf4ecd2a09b"} Nov 02 17:51:31.255562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:31.255521 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:51:31.255788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:31.255578 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 17:51:32.256648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.256177 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:51:32.256648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.256239 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 17:51:32.531850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.531720 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7529-21c899" podStartSLOduration=3.531700743 podStartE2EDuration="3.531700743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:30.92097932 +0000 UTC m=+1677.994506524" watchObservedRunningTime="2026-11-02 17:51:32.531700743 +0000 UTC m=+1679.605227948" Nov 02 17:51:32.532024 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.531981 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7529-21c899"] Nov 02 17:51:32.536393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.536362 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7529-21c899"] Nov 02 17:51:32.734391 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.734353 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs"] Nov 02 17:51:32.734629 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.734607 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" containerID="cri-o://1a8f028d1283821f1a23288491dc5bf1f4180a93026f6721ad42189321a82f1c" gracePeriod=30 Nov 02 17:51:32.797757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:32.797576 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7529-21c899" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" containerName="test-curl-7529-21c899" containerID="cri-o://f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee" gracePeriod=2 Nov 02 17:51:32.932074 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:32.932018 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-21c899\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:33.131137 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:33.131091 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-21c899\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:33.255362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:33.255326 5647 patch_prober.go:29] interesting pod/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Nov 02 17:51:33.255541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:33.255380 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Nov 02 17:51:34.260369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.260340 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:34.260825 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.260727 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:34.261677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.261657 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:51:34.675071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.675038 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9"] Nov 02 17:51:34.675838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.675765 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" containerName="test-curl-7529-21c899" Nov 02 17:51:34.675838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.675789 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" containerName="test-curl-7529-21c899" Nov 02 17:51:34.676036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.675960 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" containerName="test-curl-7529-21c899" Nov 02 17:51:34.680499 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.680469 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.702594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.702561 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9"] Nov 02 17:51:34.740595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.740560 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-700abe-openshift-default-dockercfg-8f4lt\"" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873448 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873503 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873540 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873579 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshps\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873691 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873713 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.873923 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.874009 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.874773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.874054 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.949331 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:34.949281 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3634fd1_cce3_4d09_9570_c1a16b10c24d.slice/crio-07c0b8e2ce4642e0bca2a8f931e866dd0ca919cdce3bad472483faf4ecd2a09b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ea30ba_2a57_4111_ab0e_c8b9eb458785.slice/crio-0360e6ec668a0a8b5eaa296c669fd02a06d647cd85c2d46bb1be4596e5978486\": RecentStats: unable to find data in memory cache]" Nov 02 17:51:34.989712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.989669 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.990036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.990012 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.990177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.990161 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:34.990311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.990296 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshps\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.012495 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.012541 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.018002 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.005456 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:34.997044 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.018400 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.018935 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.012726 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.020914 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.020963 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.021639 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.036548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.022260 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.069530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.068921 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshps\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.080017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.079991 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token\") pod \"e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.311091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.310474 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:35.364497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.364340 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:35.530532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.530445 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk\" (UniqueName: \"kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk\") pod \"e3634fd1-cce3-4d09-9570-c1a16b10c24d\" (UID: \"e3634fd1-cce3-4d09-9570-c1a16b10c24d\") " Nov 02 17:51:35.532796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.532767 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk" pod "e3634fd1-cce3-4d09-9570-c1a16b10c24d" (UID: "e3634fd1-cce3-4d09-9570-c1a16b10c24d"). InnerVolumeSpecName "kube-api-access-tc7kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:35.539615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.539586 5647 scope.go:122] "RemoveContainer" containerID="7f9e909be187933442ed3db0bb8f246b4ccabee4fff6c940aa697534c4eab3cf" Nov 02 17:51:35.553037 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.553009 5647 scope.go:122] "RemoveContainer" containerID="769a4b4c5b5c854854d33649a27156ea12fcb0c1948e2d411897e25392e0d437" Nov 02 17:51:35.665802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.637998 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc7kk\" (UniqueName: \"kubernetes.io/projected/e3634fd1-cce3-4d09-9570-c1a16b10c24d-kube-api-access-tc7kk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:35.865272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.857588 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:35.865272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.857766 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:35.865272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.857870 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:35.877926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.877873 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9"] Nov 02 17:51:35.914356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.914243 5647 generic.go:356] "Generic (PLEG): container finished" podID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" containerID="f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee" exitCode=137 Nov 02 17:51:35.914482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.914379 5647 scope.go:122] "RemoveContainer" containerID="f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee" Nov 02 17:51:35.914889 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.914599 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:35.926460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.926418 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" event={"ID":"9e00115b-49bd-4a20-b83b-c845f2b06343","Type":"ContainerStarted","Data":"4e6fca03e584acf1c4d7548f0809280df24a59fea506a753794f1d40f81547e5"} Nov 02 17:51:35.963028 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:35.962987 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:51:35.975915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.975719 5647 scope.go:122] "RemoveContainer" containerID="f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee" Nov 02 17:51:35.976236 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:35.976212 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee\": container with ID starting with f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee not found: ID does not exist" containerID="f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee" Nov 02 17:51:35.976340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:35.976249 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee"} err="failed to get container status \"f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee\": rpc error: code = NotFound desc = could not find container \"f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee\": container with ID starting with f1278f375efa07dc5dee6e44b8f466df09d6891874f180ee6ecdb7cfc9d3ffee not found: ID does not exist" Nov 02 17:51:36.048785 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:36.037077 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-21c899\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:36.565576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.565486 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4"] Nov 02 17:51:36.569961 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.569942 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.623638 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:36.623598 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-21c899\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:36.628090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.628057 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fa615-dockercfg-rqsfq\"" Nov 02 17:51:36.628941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.628915 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fa615\"" Nov 02 17:51:36.677835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.677802 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4"] Nov 02 17:51:36.679857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.679832 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92vbf\" (UniqueName: \"kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.680017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.679903 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.781301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.781261 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92vbf\" (UniqueName: \"kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.781487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.781318 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.782087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.782065 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.840205 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.840177 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92vbf\" (UniqueName: \"kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf\") pod \"payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.884366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.884331 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:36.938663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:36.938623 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" event={"ID":"9e00115b-49bd-4a20-b83b-c845f2b06343","Type":"ContainerStarted","Data":"4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f"} Nov 02 17:51:37.067354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.067297 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podStartSLOduration=3.067282026 podStartE2EDuration="3.067282026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:37.066152834 +0000 UTC m=+1684.139680037" watchObservedRunningTime="2026-11-02 17:51:37.067282026 +0000 UTC m=+1684.140809229" Nov 02 17:51:37.083545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.083510 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" path="/var/lib/kubelet/pods/e3634fd1-cce3-4d09-9570-c1a16b10c24d/volumes" Nov 02 17:51:37.118130 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:51:37.118046 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7529-21c899\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="e3634fd1-cce3-4d09-9570-c1a16b10c24d" pod="opendatahub/test-curl-7529-21c899" Nov 02 17:51:37.261604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.261565 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs"] Nov 02 17:51:37.268091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.268066 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.276685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.276658 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4"] Nov 02 17:51:37.277032 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:37.276996 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49445ce7_b2d9_4714_b340_2e4426ac72f6.slice/crio-6543fbaf040f9c0aae906b90f7c67e1a9ef81992f8510fa24247485502be2063 WatchSource:0}: Error finding container 6543fbaf040f9c0aae906b90f7c67e1a9ef81992f8510fa24247485502be2063: Status 404 returned error can't find the container with id 6543fbaf040f9c0aae906b90f7c67e1a9ef81992f8510fa24247485502be2063 Nov 02 17:51:37.321811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.321775 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:37.327454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.327428 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:37.339330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.339300 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs"] Nov 02 17:51:37.402332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.402246 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnddw\" (UniqueName: \"kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw\") pod \"payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.402534 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.402511 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.503644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.503604 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.503845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.503677 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnddw\" (UniqueName: \"kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw\") pod \"payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.504338 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.504316 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.562643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.562586 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnddw\" (UniqueName: \"kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw\") pod \"payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.599075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.599040 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:37.950536 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.950497 5647 generic.go:356] "Generic (PLEG): container finished" podID="752613f5-4c69-4c73-a417-93325548fc69" containerID="1a8f028d1283821f1a23288491dc5bf1f4180a93026f6721ad42189321a82f1c" exitCode=0 Nov 02 17:51:37.950700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.950526 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" event={"ID":"752613f5-4c69-4c73-a417-93325548fc69","Type":"ContainerDied","Data":"1a8f028d1283821f1a23288491dc5bf1f4180a93026f6721ad42189321a82f1c"} Nov 02 17:51:37.950700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.950611 5647 scope.go:122] "RemoveContainer" containerID="b3faf3307456bd8a2c1c578de252d61e01d452e45d55ddd16b256f80d1b688af" Nov 02 17:51:37.954553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.954510 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" event={"ID":"49445ce7-b2d9-4714-b340-2e4426ac72f6","Type":"ContainerStarted","Data":"6543fbaf040f9c0aae906b90f7c67e1a9ef81992f8510fa24247485502be2063"} Nov 02 17:51:37.954675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.954589 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:37.955683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.955660 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:51:37.980908 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:37.980856 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs"] Nov 02 17:51:37.982992 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:37.982959 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae87f4e_e00e_4e03_acb5_867a3cae39e2.slice/crio-f17f380ccf5ec321bb7c3d6c02f0cc6b6386512ce0fe69f740ee4db533c3f57f WatchSource:0}: Error finding container f17f380ccf5ec321bb7c3d6c02f0cc6b6386512ce0fe69f740ee4db533c3f57f: Status 404 returned error can't find the container with id f17f380ccf5ec321bb7c3d6c02f0cc6b6386512ce0fe69f740ee4db533c3f57f Nov 02 17:51:38.004114 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.004088 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:51:38.113282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113240 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113322 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113415 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113461 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113547 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113651 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113863 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113769 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.113920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.113871 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs\") pod \"752613f5-4c69-4c73-a417-93325548fc69\" (UID: \"752613f5-4c69-4c73-a417-93325548fc69\") " Nov 02 17:51:38.114296 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.114270 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:38.114460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.114286 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:38.114815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.114668 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:51:38.115114 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.114946 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:38.115215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.115169 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:38.115546 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.115525 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.115684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.115551 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.115684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.115566 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752613f5-4c69-4c73-a417-93325548fc69-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.115684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.115580 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.115684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.115594 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.117063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.117031 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "kube-api-access-whbqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:38.117324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.117298 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:51:38.117536 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.117508 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:51:38.117996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.117976 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token" pod "752613f5-4c69-4c73-a417-93325548fc69" (UID: "752613f5-4c69-4c73-a417-93325548fc69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:51:38.217612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.217526 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.217612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.217561 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752613f5-4c69-4c73-a417-93325548fc69-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.217612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.217578 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752613f5-4c69-4c73-a417-93325548fc69-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:38.217612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:38.217592 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbqk\" (UniqueName: \"kubernetes.io/projected/752613f5-4c69-4c73-a417-93325548fc69-kube-api-access-whbqk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:51:39.088758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:39.084844 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" Nov 02 17:51:39.099115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:39.097648 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs" event={"ID":"752613f5-4c69-4c73-a417-93325548fc69","Type":"ContainerDied","Data":"0c65d3158ed113fcc1b5cc30d7952dd5ad21321b5f4a61edd8dea20ceb37d57b"} Nov 02 17:51:39.099115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:39.097718 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" event={"ID":"bae87f4e-e00e-4e03-acb5-867a3cae39e2","Type":"ContainerStarted","Data":"f17f380ccf5ec321bb7c3d6c02f0cc6b6386512ce0fe69f740ee4db533c3f57f"} Nov 02 17:51:39.099115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:39.097772 5647 scope.go:122] "RemoveContainer" containerID="1a8f028d1283821f1a23288491dc5bf1f4180a93026f6721ad42189321a82f1c" Nov 02 17:51:39.273420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:39.273386 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs"] Nov 02 17:51:39.322930 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:39.322896 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f0f3e1-openshift-default-665997cd7c-jdbqs"] Nov 02 17:51:41.082700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:41.082665 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752613f5-4c69-4c73-a417-93325548fc69" path="/var/lib/kubelet/pods/752613f5-4c69-4c73-a417-93325548fc69/volumes" Nov 02 17:51:41.140011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:41.139952 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" event={"ID":"49445ce7-b2d9-4714-b340-2e4426ac72f6","Type":"ContainerStarted","Data":"d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f"} Nov 02 17:51:41.140178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:41.140126 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:41.142954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:41.142924 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" event={"ID":"bae87f4e-e00e-4e03-acb5-867a3cae39e2","Type":"ContainerStarted","Data":"9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77"} Nov 02 17:51:41.143149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:41.143029 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:41.277310 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:41.277248 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" podStartSLOduration=2.368415574 podStartE2EDuration="5.277232475s" totalImagesPullingTime="2.908816901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:41.274141829 +0000 UTC m=+1688.347669033" watchObservedRunningTime="2026-11-02 17:51:41.277232475 +0000 UTC m=+1688.350759677" Nov 02 17:51:48.769293 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.769222 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" podStartSLOduration=9.204295095 podStartE2EDuration="11.769205841s" totalImagesPullingTime="2.564910746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:41.435128024 +0000 UTC m=+1688.508655225" watchObservedRunningTime="2026-11-02 17:51:48.769205841 +0000 UTC m=+1695.842733044" Nov 02 17:51:48.771129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771102 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc"] Nov 02 17:51:48.771813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771795 5647 cpu_manager.go:414] "containerMap: removing container" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" Nov 02 17:51:48.771813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771811 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" Nov 02 17:51:48.771965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771822 5647 cpu_manager.go:414] "containerMap: removing container" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" Nov 02 17:51:48.771965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771828 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" Nov 02 17:51:48.771965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771936 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" Nov 02 17:51:48.771965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.771944 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="752613f5-4c69-4c73-a417-93325548fc69" containerName="istio-proxy" Nov 02 17:51:48.777123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.777101 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.805619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.805586 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.805827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.805644 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.805827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.805663 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.805827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.805774 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.806032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.805852 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.806032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.805959 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxtkq\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.806151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.806042 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.806151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.806139 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.806262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.806184 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908028 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908078 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908144 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908187 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908206 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908233 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908271 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908340 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxtkq\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908451 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908505 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908562 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908614 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.908876 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.908848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.909115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.909095 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.910976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.910949 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.911412 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.911392 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.980995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.980961 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.981308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.981286 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxtkq\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:48.999428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:48.999395 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc"] Nov 02 17:51:49.092897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:49.092868 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:49.572040 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:49.572015 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc"] Nov 02 17:51:49.572401 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:49.572378 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76250067_6ed8_4158_8363_30bac127a046.slice/crio-65f7eaf5af05295aa6b5815e9c90a3270ed7b253fb66783c83e96ec621137f89 WatchSource:0}: Error finding container 65f7eaf5af05295aa6b5815e9c90a3270ed7b253fb66783c83e96ec621137f89: Status 404 returned error can't find the container with id 65f7eaf5af05295aa6b5815e9c90a3270ed7b253fb66783c83e96ec621137f89 Nov 02 17:51:49.575344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:49.575309 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:49.575436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:49.575375 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:49.575436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:49.575413 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:50.223822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.223698 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" event={"ID":"76250067-6ed8-4158-8363-30bac127a046","Type":"ContainerStarted","Data":"3b44dd15c882bbaa72b9f328106ad8922af3de1a30756a15d43aef6ed3399fac"} Nov 02 17:51:50.223822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.223797 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" event={"ID":"76250067-6ed8-4158-8363-30bac127a046","Type":"ContainerStarted","Data":"65f7eaf5af05295aa6b5815e9c90a3270ed7b253fb66783c83e96ec621137f89"} Nov 02 17:51:50.225662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.225632 5647 generic.go:356] "Generic (PLEG): container finished" podID="a0753785-7233-4533-a398-0059cd0b8758" containerID="9aababaca92a01c536307e4c65145139419eac16cfe4ed272f9e8957311c9d13" exitCode=0 Nov 02 17:51:50.225841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.225711 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" event={"ID":"a0753785-7233-4533-a398-0059cd0b8758","Type":"ContainerDied","Data":"9aababaca92a01c536307e4c65145139419eac16cfe4ed272f9e8957311c9d13"} Nov 02 17:51:50.226178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.226162 5647 scope.go:122] "RemoveContainer" containerID="9aababaca92a01c536307e4c65145139419eac16cfe4ed272f9e8957311c9d13" Nov 02 17:51:50.228159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.228121 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:50.228273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.228213 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:50.228314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.228265 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:50.438595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.438541 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podStartSLOduration=2.438525776 podStartE2EDuration="2.438525776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:50.399252098 +0000 UTC m=+1697.472779313" watchObservedRunningTime="2026-11-02 17:51:50.438525776 +0000 UTC m=+1697.512052978" Nov 02 17:51:50.618764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:50.615681 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:51:51.093929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.093898 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:51.095347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.095325 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 17:51:51.095432 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.095372 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 17:51:51.268290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.268251 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" event={"ID":"a0753785-7233-4533-a398-0059cd0b8758","Type":"ContainerStarted","Data":"53d39cb52331f7bae20e20f9736129a5701a21a4fb4c5d2c49de290c445de38e"} Nov 02 17:51:51.299770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.299726 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:51:51.301005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.300978 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 17:51:51.301145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:51.301025 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 17:51:52.093363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:52.093327 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 17:51:52.093611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:52.093380 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 17:51:52.154758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:52.153909 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:51:52.154959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:52.154934 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:51:52.300407 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:52.300373 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 17:51:52.300873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:52.300437 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 17:51:53.093249 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:53.093216 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 17:51:53.093441 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:53.093271 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 17:51:53.299296 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:53.299260 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 17:51:53.299478 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:53.299308 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 17:51:54.093783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:54.093724 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.094198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:54.093800 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 17:51:54.307758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:54.299538 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 17:51:54.307758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:54.299614 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 17:51:55.093268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:55.093189 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.093486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:55.093466 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Nov 02 17:51:55.299295 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:55.299263 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 17:51:55.299726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:55.299311 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 17:51:56.173316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:56.173282 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:56.174087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:56.174067 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:56.174783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:56.174758 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:51:56.299450 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:56.299421 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Nov 02 17:51:56.299859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:56.299467 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Nov 02 17:51:56.484567 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:56.484485 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:51:57.304587 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:57.304559 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:51:57.305022 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:57.304856 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:51:57.305592 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:57.305572 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:51:58.798724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.798675 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn"] Nov 02 17:51:58.818279 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.818246 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.887763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.880105 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f10385-openshift-default-dockercfg-4xs5v\"" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.946963 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947038 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947123 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzcd\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947161 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947184 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947237 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947262 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947391 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:58.966814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:58.947425 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.026931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.026896 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn"] Nov 02 17:51:59.058093 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.058005 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzcd\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.058362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.058335 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.058496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.058480 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.058657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.058640 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.058810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.058791 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.059030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.059010 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.059173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.059157 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.059489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.059467 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.059648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.059633 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.060992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.060971 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.063853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.061945 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.064111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.062199 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.064301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.062567 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.064454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.063434 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.170003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.166270 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzcd\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.170003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.167406 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.170003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.167535 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.170003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.169771 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo\") pod \"e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.468137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.468102 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:51:59.968343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.968319 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn"] Nov 02 17:51:59.968945 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:51:59.968921 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01ea49e_000a_40ec_b40c_d461cc102fbf.slice/crio-0e7c2fcfb279a68664aff7af8c5323e6a891532263b6836892c976e91a5764d6 WatchSource:0}: Error finding container 0e7c2fcfb279a68664aff7af8c5323e6a891532263b6836892c976e91a5764d6: Status 404 returned error can't find the container with id 0e7c2fcfb279a68664aff7af8c5323e6a891532263b6836892c976e91a5764d6 Nov 02 17:51:59.971603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.971568 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:59.971709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.971637 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:51:59.971709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:51:59.971667 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:00.403814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:00.403774 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" event={"ID":"d01ea49e-000a-40ec-b40c-d461cc102fbf","Type":"ContainerStarted","Data":"b940f52dd8abe7e55bc5453b4d82498a2165bbb72f2226655e5df951723273b5"} Nov 02 17:52:00.404028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:00.403847 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" event={"ID":"d01ea49e-000a-40ec-b40c-d461cc102fbf","Type":"ContainerStarted","Data":"0e7c2fcfb279a68664aff7af8c5323e6a891532263b6836892c976e91a5764d6"} Nov 02 17:52:00.468389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:00.468354 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:52:00.535812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:00.535759 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podStartSLOduration=2.535725018 podStartE2EDuration="2.535725018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:00.533340518 +0000 UTC m=+1707.606867759" watchObservedRunningTime="2026-11-02 17:52:00.535725018 +0000 UTC m=+1707.609252222" Nov 02 17:52:01.473892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:01.473859 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:52:02.417721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:02.417681 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:52:02.418858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:02.418836 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:52:05.947939 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:05.947897 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:52:11.530322 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:11.530236 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l"] Nov 02 17:52:11.677958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:11.677919 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn"] Nov 02 17:52:16.531577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.531497 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" podUID="d39e0598-ceb8-4d1b-b992-006710906761" containerName="payload-processing" containerID="cri-o://85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37" gracePeriod=25 Nov 02 17:52:16.679301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.679252 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" podUID="27198416-6510-478e-b361-855d53ff3e40" containerName="payload-pre-processing" containerID="cri-o://4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5" gracePeriod=25 Nov 02 17:52:16.796065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.796033 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr"] Nov 02 17:52:16.824556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.824531 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:52:16.897001 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.896963 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume\") pod \"d39e0598-ceb8-4d1b-b992-006710906761\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " Nov 02 17:52:16.897177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.897051 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b\" (UniqueName: \"kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b\") pod \"d39e0598-ceb8-4d1b-b992-006710906761\" (UID: \"d39e0598-ceb8-4d1b-b992-006710906761\") " Nov 02 17:52:16.899189 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.899154 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume" pod "d39e0598-ceb8-4d1b-b992-006710906761" (UID: "d39e0598-ceb8-4d1b-b992-006710906761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:16.902810 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.902561 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b" pod "d39e0598-ceb8-4d1b-b992-006710906761" (UID: "d39e0598-ceb8-4d1b-b992-006710906761"). InnerVolumeSpecName "kube-api-access-25g6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:16.974847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:16.974820 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:52:17.000374 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.000335 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39e0598-ceb8-4d1b-b992-006710906761-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:17.000374 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.000372 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25g6b\" (UniqueName: \"kubernetes.io/projected/d39e0598-ceb8-4d1b-b992-006710906761-kube-api-access-25g6b\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:17.102170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.101056 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr\" (UniqueName: \"kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr\") pod \"27198416-6510-478e-b361-855d53ff3e40\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " Nov 02 17:52:17.102170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.101196 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume\") pod \"27198416-6510-478e-b361-855d53ff3e40\" (UID: \"27198416-6510-478e-b361-855d53ff3e40\") " Nov 02 17:52:17.102928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.102565 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume" pod "27198416-6510-478e-b361-855d53ff3e40" (UID: "27198416-6510-478e-b361-855d53ff3e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:17.106084 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.106040 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr" pod "27198416-6510-478e-b361-855d53ff3e40" (UID: "27198416-6510-478e-b361-855d53ff3e40"). InnerVolumeSpecName "kube-api-access-dtwqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:17.236758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.215578 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtwqr\" (UniqueName: \"kubernetes.io/projected/27198416-6510-478e-b361-855d53ff3e40-kube-api-access-dtwqr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:17.236758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.215623 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27198416-6510-478e-b361-855d53ff3e40-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:17.578018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.577982 5647 generic.go:356] "Generic (PLEG): container finished" podID="d39e0598-ceb8-4d1b-b992-006710906761" containerID="85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37" exitCode=0 Nov 02 17:52:17.578504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.578041 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" Nov 02 17:52:17.578504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.578069 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" event={"ID":"d39e0598-ceb8-4d1b-b992-006710906761","Type":"ContainerDied","Data":"85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37"} Nov 02 17:52:17.578504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.578134 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l" event={"ID":"d39e0598-ceb8-4d1b-b992-006710906761","Type":"ContainerDied","Data":"df08db65211c391c3fa1143b0bfe7995cb8b12e03665a3a43cdfc40e4806f451"} Nov 02 17:52:17.578504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.578161 5647 scope.go:122] "RemoveContainer" containerID="85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37" Nov 02 17:52:17.580089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.579905 5647 generic.go:356] "Generic (PLEG): container finished" podID="27198416-6510-478e-b361-855d53ff3e40" containerID="4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5" exitCode=0 Nov 02 17:52:17.580089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.579949 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" event={"ID":"27198416-6510-478e-b361-855d53ff3e40","Type":"ContainerDied","Data":"4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5"} Nov 02 17:52:17.580089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.579965 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" Nov 02 17:52:17.580089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.579995 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn" event={"ID":"27198416-6510-478e-b361-855d53ff3e40","Type":"ContainerDied","Data":"fac44be08f5a5da2dca7017cd4479e87569653a4bfc10ffc54761824ff736872"} Nov 02 17:52:17.591896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.591877 5647 scope.go:122] "RemoveContainer" containerID="85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37" Nov 02 17:52:17.592222 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:17.592203 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37\": container with ID starting with 85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37 not found: ID does not exist" containerID="85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37" Nov 02 17:52:17.592284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.592231 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37"} err="failed to get container status \"85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37\": rpc error: code = NotFound desc = could not find container \"85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37\": container with ID starting with 85a7c00edb2386e5cc24924b9138507c98f75f917b470fb21518245b11d21b37 not found: ID does not exist" Nov 02 17:52:17.592284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.592251 5647 scope.go:122] "RemoveContainer" containerID="4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5" Nov 02 17:52:17.603226 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.603201 5647 scope.go:122] "RemoveContainer" containerID="4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5" Nov 02 17:52:17.603548 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:17.603525 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5\": container with ID starting with 4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5 not found: ID does not exist" containerID="4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5" Nov 02 17:52:17.603644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.603555 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5"} err="failed to get container status \"4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5\": rpc error: code = NotFound desc = could not find container \"4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5\": container with ID starting with 4ca8ba829005d9127b907ddedaa6981fa6f708ce474caa90faf2df9382d9bef5 not found: ID does not exist" Nov 02 17:52:17.750071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.750033 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn"] Nov 02 17:52:17.752955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.752929 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e61288-5d5698fd8c-ltntn"] Nov 02 17:52:17.863269 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.863240 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l"] Nov 02 17:52:17.869362 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:17.869334 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e61288-6dd797b9d7-5ct9l"] Nov 02 17:52:18.999235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:18.999204 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7"] Nov 02 17:52:19.000123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.000099 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d39e0598-ceb8-4d1b-b992-006710906761" containerName="payload-processing" Nov 02 17:52:19.000250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.000123 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39e0598-ceb8-4d1b-b992-006710906761" containerName="payload-processing" Nov 02 17:52:19.000250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.000168 5647 cpu_manager.go:414] "containerMap: removing container" podUID="27198416-6510-478e-b361-855d53ff3e40" containerName="payload-pre-processing" Nov 02 17:52:19.000250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.000176 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27198416-6510-478e-b361-855d53ff3e40" containerName="payload-pre-processing" Nov 02 17:52:19.000411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.000306 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="27198416-6510-478e-b361-855d53ff3e40" containerName="payload-pre-processing" Nov 02 17:52:19.000411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.000332 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39e0598-ceb8-4d1b-b992-006710906761" containerName="payload-processing" Nov 02 17:52:19.003990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.003967 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.084963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.084926 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27198416-6510-478e-b361-855d53ff3e40" path="/var/lib/kubelet/pods/27198416-6510-478e-b361-855d53ff3e40/volumes" Nov 02 17:52:19.085527 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.085503 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39e0598-ceb8-4d1b-b992-006710906761" path="/var/lib/kubelet/pods/d39e0598-ceb8-4d1b-b992-006710906761/volumes" Nov 02 17:52:19.166649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.166614 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.166898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.166661 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.166898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.166800 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7h79\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.166991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.166908 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.167034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.167014 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.167075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.167046 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.167075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.167067 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.167209 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.167186 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.167267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.167254 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.214139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.214105 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7"] Nov 02 17:52:19.268415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268329 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268381 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268417 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268458 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268506 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268632 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268676 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268774 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268718 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7h79\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268773 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268794 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.268919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268888 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.269091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.268948 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.269091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.269038 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.269326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.269247 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.271052 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.271028 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.271414 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.271397 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.330256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.330212 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.330413 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.330258 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7h79\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79\") pod \"e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.618054 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.618018 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:19.949616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.949528 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9"] Nov 02 17:52:19.949817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:19.949776 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" podUID="047c0850-a124-4e4e-ac98-380c15f7e971" containerName="maas-api" containerID="cri-o://6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2" gracePeriod=30 Nov 02 17:52:20.116844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.116799 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7"] Nov 02 17:52:20.118045 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:52:20.118013 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bdb11b_0842_4c0c_a3e6_8f61d3aa1de8.slice/crio-c20a90cfe2c53377abc90004cdb8bf8fad8d89e6c67d97920dba15ef36e13f41 WatchSource:0}: Error finding container c20a90cfe2c53377abc90004cdb8bf8fad8d89e6c67d97920dba15ef36e13f41: Status 404 returned error can't find the container with id c20a90cfe2c53377abc90004cdb8bf8fad8d89e6c67d97920dba15ef36e13f41 Nov 02 17:52:20.121538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.121499 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:20.121660 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.121573 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:20.121660 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.121619 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:20.410760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.408077 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:52:20.513986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.511448 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs\") pod \"047c0850-a124-4e4e-ac98-380c15f7e971\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " Nov 02 17:52:20.513986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.512103 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\") pod \"047c0850-a124-4e4e-ac98-380c15f7e971\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " Nov 02 17:52:20.526068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.519935 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca\") pod \"047c0850-a124-4e4e-ac98-380c15f7e971\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " Nov 02 17:52:20.526068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.520302 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca" pod "047c0850-a124-4e4e-ac98-380c15f7e971" (UID: "047c0850-a124-4e4e-ac98-380c15f7e971"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:20.526068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.520886 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs" pod "047c0850-a124-4e4e-ac98-380c15f7e971" (UID: "047c0850-a124-4e4e-ac98-380c15f7e971"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:20.526068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.521093 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7\" (UniqueName: \"kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7\") pod \"047c0850-a124-4e4e-ac98-380c15f7e971\" (UID: \"047c0850-a124-4e4e-ac98-380c15f7e971\") " Nov 02 17:52:20.526068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.522763 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:20.526068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.522797 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/047c0850-a124-4e4e-ac98-380c15f7e971-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:20.538232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.532478 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls" pod "047c0850-a124-4e4e-ac98-380c15f7e971" (UID: "047c0850-a124-4e4e-ac98-380c15f7e971"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:52:20.538232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.532612 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7" pod "047c0850-a124-4e4e-ac98-380c15f7e971" (UID: "047c0850-a124-4e4e-ac98-380c15f7e971"). InnerVolumeSpecName "kube-api-access-ln7z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:20.624427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.624392 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7z7\" (UniqueName: \"kubernetes.io/projected/047c0850-a124-4e4e-ac98-380c15f7e971-kube-api-access-ln7z7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:20.624427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.624425 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/047c0850-a124-4e4e-ac98-380c15f7e971-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:20.688365 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.688334 5647 generic.go:356] "Generic (PLEG): container finished" podID="047c0850-a124-4e4e-ac98-380c15f7e971" containerID="6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2" exitCode=0 Nov 02 17:52:20.688559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.688400 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" Nov 02 17:52:20.688559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.688430 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" event={"ID":"047c0850-a124-4e4e-ac98-380c15f7e971","Type":"ContainerDied","Data":"6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2"} Nov 02 17:52:20.688559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.688498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9" event={"ID":"047c0850-a124-4e4e-ac98-380c15f7e971","Type":"ContainerDied","Data":"8a3514a1b9cace33b71efea41fb311bf73efabde5f9bc8b5303449bc6942f396"} Nov 02 17:52:20.688559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.688516 5647 scope.go:122] "RemoveContainer" containerID="6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2" Nov 02 17:52:20.690764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.690716 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" event={"ID":"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8","Type":"ContainerStarted","Data":"f19e37e25618c4b65c25d8a3d421cfed1ba10dde4b68ff8e67f41ba954812d5b"} Nov 02 17:52:20.690882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.690781 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" event={"ID":"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8","Type":"ContainerStarted","Data":"c20a90cfe2c53377abc90004cdb8bf8fad8d89e6c67d97920dba15ef36e13f41"} Nov 02 17:52:20.699143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.699123 5647 scope.go:122] "RemoveContainer" containerID="6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2" Nov 02 17:52:20.699511 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:20.699488 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2\": container with ID starting with 6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2 not found: ID does not exist" containerID="6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2" Nov 02 17:52:20.699623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.699520 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2"} err="failed to get container status \"6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2\": rpc error: code = NotFound desc = could not find container \"6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2\": container with ID starting with 6b9b9e5f4b67f585e7803063b724fb8da2dfc3596f2ea1953ac05b46ff980bf2 not found: ID does not exist" Nov 02 17:52:20.858364 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.858335 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9"] Nov 02 17:52:20.863110 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:20.863080 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8cbedb9f-65d56d9c6-jpbr9"] Nov 02 17:52:21.022761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.012515 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podStartSLOduration=3.012490562 podStartE2EDuration="3.012490562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:20.983885668 +0000 UTC m=+1728.057412873" watchObservedRunningTime="2026-11-02 17:52:21.012490562 +0000 UTC m=+1728.086017776" Nov 02 17:52:21.119816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.116372 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="047c0850-a124-4e4e-ac98-380c15f7e971" path="/var/lib/kubelet/pods/047c0850-a124-4e4e-ac98-380c15f7e971/volumes" Nov 02 17:52:21.630762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.619177 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:21.630762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.623050 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 17:52:21.630762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.623106 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 17:52:21.634185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.634152 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz"] Nov 02 17:52:21.749843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.749804 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr"] Nov 02 17:52:21.797335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:21.797297 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" podUID="a30e5bbd-9f3d-44db-9df3-233a53d92b01" containerName="payload-processing" containerID="cri-o://90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33" gracePeriod=25 Nov 02 17:52:22.271178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.270807 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:52:22.447854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.447459 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q"] Nov 02 17:52:22.447854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.447494 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt\" (UniqueName: \"kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt\") pod \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " Nov 02 17:52:22.447854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.447631 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume\") pod \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\" (UID: \"a30e5bbd-9f3d-44db-9df3-233a53d92b01\") " Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448384 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume" pod "a30e5bbd-9f3d-44db-9df3-233a53d92b01" (UID: "a30e5bbd-9f3d-44db-9df3-233a53d92b01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448417 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a30e5bbd-9f3d-44db-9df3-233a53d92b01" containerName="payload-processing" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448436 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30e5bbd-9f3d-44db-9df3-233a53d92b01" containerName="payload-processing" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448459 5647 cpu_manager.go:414] "containerMap: removing container" podUID="047c0850-a124-4e4e-ac98-380c15f7e971" containerName="maas-api" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448467 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="047c0850-a124-4e4e-ac98-380c15f7e971" containerName="maas-api" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448650 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30e5bbd-9f3d-44db-9df3-233a53d92b01" containerName="payload-processing" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448664 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="047c0850-a124-4e4e-ac98-380c15f7e971" containerName="maas-api" Nov 02 17:52:22.448787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.448786 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a30e5bbd-9f3d-44db-9df3-233a53d92b01-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:22.451390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.451357 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt" pod "a30e5bbd-9f3d-44db-9df3-233a53d92b01" (UID: "a30e5bbd-9f3d-44db-9df3-233a53d92b01"). InnerVolumeSpecName "kube-api-access-b2kbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:22.452123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.452104 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.549075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.543439 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q"] Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.553644 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.553708 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.553774 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mz92\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.553852 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.553909 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.553967 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.554046 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.554077 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.554136 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.556268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.554347 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2kbt\" (UniqueName: \"kubernetes.io/projected/a30e5bbd-9f3d-44db-9df3-233a53d92b01-kube-api-access-b2kbt\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:22.560129 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.557965 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-666de0-openshift-default-dockercfg-cwt5r\"" Nov 02 17:52:22.633909 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.633378 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 17:52:22.633909 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.633476 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 17:52:22.655389 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655357 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655414 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655449 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655490 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655648 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655674 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655833 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mz92\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655910 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.655997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655979 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.656092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.655978 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.656273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.656246 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.656395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.656248 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.656395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.656290 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.656686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.656400 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.658248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.658222 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.658636 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.658618 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.755806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.750377 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.755806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.753081 5647 generic.go:356] "Generic (PLEG): container finished" podID="a30e5bbd-9f3d-44db-9df3-233a53d92b01" containerID="90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33" exitCode=0 Nov 02 17:52:22.755806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.754189 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" Nov 02 17:52:22.756103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.755840 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" event={"ID":"a30e5bbd-9f3d-44db-9df3-233a53d92b01","Type":"ContainerDied","Data":"90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33"} Nov 02 17:52:22.756103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.755887 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr" event={"ID":"a30e5bbd-9f3d-44db-9df3-233a53d92b01","Type":"ContainerDied","Data":"1bec9200afdf5f2456936d16c639d795e8b88065fcb22d83fa529414d3669019"} Nov 02 17:52:22.756103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.755912 5647 scope.go:122] "RemoveContainer" containerID="90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33" Nov 02 17:52:22.775030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.774962 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mz92\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92\") pod \"e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:22.895880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:22.895300 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:23.051055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.050254 5647 scope.go:122] "RemoveContainer" containerID="90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33" Nov 02 17:52:23.052837 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:23.052723 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33\": container with ID starting with 90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33 not found: ID does not exist" containerID="90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33" Nov 02 17:52:23.052837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.052798 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33"} err="failed to get container status \"90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33\": rpc error: code = NotFound desc = could not find container \"90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33\": container with ID starting with 90276dea0ea990adaffbe89bac6b79b7cc3c54b9f23ae382a21e5bab5f6d6e33 not found: ID does not exist" Nov 02 17:52:23.060997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.060889 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr"] Nov 02 17:52:23.062078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.061799 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8cbedb9f-6c65d96c65-xj7gr"] Nov 02 17:52:23.083512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.083477 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30e5bbd-9f3d-44db-9df3-233a53d92b01" path="/var/lib/kubelet/pods/a30e5bbd-9f3d-44db-9df3-233a53d92b01/volumes" Nov 02 17:52:23.555115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.533570 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q"] Nov 02 17:52:23.637761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.619380 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:23.637761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.619488 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:23.637761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.619533 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:23.637761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.625097 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 17:52:23.637761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.625153 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 17:52:23.898193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:23.898117 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" event={"ID":"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9","Type":"ContainerStarted","Data":"fba86cec35bb675afe45603300943bbdb92cd4c9c6e2ebce8c9deaaf8759b465"} Nov 02 17:52:24.626089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:24.626051 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 17:52:24.628635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:24.628597 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 17:52:24.996156 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:24.996054 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" event={"ID":"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9","Type":"ContainerStarted","Data":"109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115"} Nov 02 17:52:25.094019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.093980 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd"] Nov 02 17:52:25.094198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.094061 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv"] Nov 02 17:52:25.094198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.094094 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv"] Nov 02 17:52:25.094315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.094288 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" containerID="cri-o://07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285" gracePeriod=30 Nov 02 17:52:25.094376 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.094346 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" containerID="cri-o://fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b" gracePeriod=30 Nov 02 17:52:25.094499 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.094472 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" containerID="cri-o://39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17" gracePeriod=30 Nov 02 17:52:25.095468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.095444 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm"] Nov 02 17:52:25.095913 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.095845 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" containerID="cri-o://45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b" gracePeriod=30 Nov 02 17:52:25.273841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.268773 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podStartSLOduration=3.2687495970000002 podStartE2EDuration="3.268749597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:25.229490428 +0000 UTC m=+1732.303017631" watchObservedRunningTime="2026-11-02 17:52:25.268749597 +0000 UTC m=+1732.342276796" Nov 02 17:52:25.631301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.631255 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 17:52:25.631781 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.631328 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 17:52:25.895707 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.895619 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:25.906766 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:25.901801 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:26.030101 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.029419 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:26.031411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.031385 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:52:26.332795 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.332704 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf"] Nov 02 17:52:26.337243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.337216 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.376688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.376653 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf"] Nov 02 17:52:26.392751 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.392699 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5bf666-openshift-default-dockercfg-zvl5p\"" Nov 02 17:52:26.553212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.553173 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.553557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.553535 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nch8f\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.553814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.553796 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.553987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.553970 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.581872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.581831 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.582207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.582182 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.582395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.582379 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.582618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.582601 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.582770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.582752 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.620794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.620751 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 02 17:52:26.622192 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.621120 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 02 17:52:26.643095 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.642879 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" podUID="884cc241-6b33-4240-8f51-3ac590b01b39" containerName="payload-processing" containerID="cri-o://00396f18f98495a1bdd62731fc02c37c507898f5062924c17fb46f5b9fafaa89" gracePeriod=25 Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.687934 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688020 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688050 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688161 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688245 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688345 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688378 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688467 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.688518 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nch8f\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.689329 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.690021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.689587 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.694656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.694544 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.695380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.695340 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.702118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.702062 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.712530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.712453 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.714060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.713973 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.750876 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.750590 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" podUID="d2112269-cd3d-4659-8156-9380e128dbde" containerName="payload-pre-processing" containerID="cri-o://066fc7d9a2322349a3f67faa3e6c993b2f1b8a44c9f7df4f37f146e650fcc8d3" gracePeriod=25 Nov 02 17:52:26.826872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.780321 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nch8f\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.857479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.857042 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token\") pod \"e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:26.955426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:26.954226 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:27.050100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.050057 5647 generic.go:356] "Generic (PLEG): container finished" podID="884cc241-6b33-4240-8f51-3ac590b01b39" containerID="00396f18f98495a1bdd62731fc02c37c507898f5062924c17fb46f5b9fafaa89" exitCode=0 Nov 02 17:52:27.050288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.050226 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" event={"ID":"884cc241-6b33-4240-8f51-3ac590b01b39","Type":"ContainerDied","Data":"00396f18f98495a1bdd62731fc02c37c507898f5062924c17fb46f5b9fafaa89"} Nov 02 17:52:27.054289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.054249 5647 generic.go:356] "Generic (PLEG): container finished" podID="d2112269-cd3d-4659-8156-9380e128dbde" containerID="066fc7d9a2322349a3f67faa3e6c993b2f1b8a44c9f7df4f37f146e650fcc8d3" exitCode=0 Nov 02 17:52:27.054755 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.054707 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" event={"ID":"d2112269-cd3d-4659-8156-9380e128dbde","Type":"ContainerDied","Data":"066fc7d9a2322349a3f67faa3e6c993b2f1b8a44c9f7df4f37f146e650fcc8d3"} Nov 02 17:52:27.069770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.069435 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:52:27.209760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.208276 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2\" (UniqueName: \"kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2\") pod \"884cc241-6b33-4240-8f51-3ac590b01b39\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " Nov 02 17:52:27.209760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.208530 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume\") pod \"884cc241-6b33-4240-8f51-3ac590b01b39\" (UID: \"884cc241-6b33-4240-8f51-3ac590b01b39\") " Nov 02 17:52:27.243112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.243075 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume" pod "884cc241-6b33-4240-8f51-3ac590b01b39" (UID: "884cc241-6b33-4240-8f51-3ac590b01b39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:27.244792 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.244706 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2" pod "884cc241-6b33-4240-8f51-3ac590b01b39" (UID: "884cc241-6b33-4240-8f51-3ac590b01b39"). InnerVolumeSpecName "kube-api-access-6vcb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:27.250212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.250185 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:52:27.330769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.330347 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vcb2\" (UniqueName: \"kubernetes.io/projected/884cc241-6b33-4240-8f51-3ac590b01b39-kube-api-access-6vcb2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:27.330769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.330397 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884cc241-6b33-4240-8f51-3ac590b01b39-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:27.431702 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.431595 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume\") pod \"d2112269-cd3d-4659-8156-9380e128dbde\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " Nov 02 17:52:27.431926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.431725 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9\" (UniqueName: \"kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9\") pod \"d2112269-cd3d-4659-8156-9380e128dbde\" (UID: \"d2112269-cd3d-4659-8156-9380e128dbde\") " Nov 02 17:52:27.432512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.432485 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume" pod "d2112269-cd3d-4659-8156-9380e128dbde" (UID: "d2112269-cd3d-4659-8156-9380e128dbde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:27.434562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.434530 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9" pod "d2112269-cd3d-4659-8156-9380e128dbde" (UID: "d2112269-cd3d-4659-8156-9380e128dbde"). InnerVolumeSpecName "kube-api-access-8qpr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:27.479229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.479196 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf"] Nov 02 17:52:27.480394 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:52:27.480361 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10786ae4_769f_4e4e_a01a_e0354e2366bd.slice/crio-51e2d9f1354b38e5d02652edd02a179a809c8b3e4b6acc57daee67c5c7b1639b WatchSource:0}: Error finding container 51e2d9f1354b38e5d02652edd02a179a809c8b3e4b6acc57daee67c5c7b1639b: Status 404 returned error can't find the container with id 51e2d9f1354b38e5d02652edd02a179a809c8b3e4b6acc57daee67c5c7b1639b Nov 02 17:52:27.483068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.483034 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:27.483140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.483102 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:27.483140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.483130 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:52:27.533916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.533888 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2112269-cd3d-4659-8156-9380e128dbde-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:27.534057 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.533918 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpr9\" (UniqueName: \"kubernetes.io/projected/d2112269-cd3d-4659-8156-9380e128dbde-kube-api-access-8qpr9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:27.922242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.922206 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:27.922816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.922790 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:27.923983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:27.923959 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:52:28.062990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.062905 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" Nov 02 17:52:28.062990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.062938 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr" event={"ID":"d2112269-cd3d-4659-8156-9380e128dbde","Type":"ContainerDied","Data":"f48995a6d947d8a93ee6775dc899b83daa4dff2a6b9e0ba0f0bb57dcd0497df6"} Nov 02 17:52:28.063218 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.063013 5647 scope.go:122] "RemoveContainer" containerID="066fc7d9a2322349a3f67faa3e6c993b2f1b8a44c9f7df4f37f146e650fcc8d3" Nov 02 17:52:28.064854 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.064825 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" Nov 02 17:52:28.064962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.064850 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz" event={"ID":"884cc241-6b33-4240-8f51-3ac590b01b39","Type":"ContainerDied","Data":"dde22b81b9c2424ba2dfc724738783d31d24d210ef19e739013e339e9eceedb1"} Nov 02 17:52:28.066880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.066848 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" event={"ID":"10786ae4-769f-4e4e-a01a-e0354e2366bd","Type":"ContainerStarted","Data":"aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9"} Nov 02 17:52:28.067029 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.066905 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" event={"ID":"10786ae4-769f-4e4e-a01a-e0354e2366bd","Type":"ContainerStarted","Data":"51e2d9f1354b38e5d02652edd02a179a809c8b3e4b6acc57daee67c5c7b1639b"} Nov 02 17:52:28.074622 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.074581 5647 scope.go:122] "RemoveContainer" containerID="00396f18f98495a1bdd62731fc02c37c507898f5062924c17fb46f5b9fafaa89" Nov 02 17:52:28.297451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.297416 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz"] Nov 02 17:52:28.301011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.300984 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-5bf4dbd4bc-b9llz"] Nov 02 17:52:28.410245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.410209 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr"] Nov 02 17:52:28.413170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.413145 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-6bb444ddc-68vfr"] Nov 02 17:52:28.538493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.538419 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podStartSLOduration=2.538399117 podStartE2EDuration="2.538399117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:28.537818163 +0000 UTC m=+1735.611345367" watchObservedRunningTime="2026-11-02 17:52:28.538399117 +0000 UTC m=+1735.611926321" Nov 02 17:52:28.954463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.954430 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:28.960226 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:28.960196 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:29.083792 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:29.083676 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884cc241-6b33-4240-8f51-3ac590b01b39" path="/var/lib/kubelet/pods/884cc241-6b33-4240-8f51-3ac590b01b39/volumes" Nov 02 17:52:29.084352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:29.084333 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2112269-cd3d-4659-8156-9380e128dbde" path="/var/lib/kubelet/pods/d2112269-cd3d-4659-8156-9380e128dbde/volumes" Nov 02 17:52:29.084652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:29.084630 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:29.084832 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:29.084709 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:52:30.423867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.423839 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:52:30.489812 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.489783 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:52:30.494722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.494697 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:52:30.527603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527576 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.527727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527635 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.527727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527692 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.527868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527772 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.527868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527831 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.527868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527864 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.528030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.527902 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.528092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.528019 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.528277 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.528250 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.528390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.528280 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:30.528634 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.528592 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.528991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.528662 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs\") pod \"f070449e-b50d-405b-abeb-b251576ed8de\" (UID: \"f070449e-b50d-405b-abeb-b251576ed8de\") " Nov 02 17:52:30.529177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.529112 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.529521 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.529494 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.529615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.529522 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.529615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.529537 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f070449e-b50d-405b-abeb-b251576ed8de-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.529615 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.529551 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.529826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.529641 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.530033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.530012 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "kube-api-access-kdvs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.530495 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.530468 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:30.530577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.530510 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.530644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.530626 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy" pod "f070449e-b50d-405b-abeb-b251576ed8de" (UID: "f070449e-b50d-405b-abeb-b251576ed8de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.536839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.536800 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:52:30.630595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630549 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.630886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630637 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.630886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630675 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.630886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630716 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.630886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630827 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630896 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630934 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.630969 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631021 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631060 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631101 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.631206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631186 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631233 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631250 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631281 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631344 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631381 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631421 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631456 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631491 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631515 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631556 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.631570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631556 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631600 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631666 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631695 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631723 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy\") pod \"3375acde-3ce8-442a-9718-e7a6799e9722\" (UID: \"3375acde-3ce8-442a-9718-e7a6799e9722\") " Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631787 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy\") pod \"2d228d91-771d-42af-8750-ce9b823760ed\" (UID: \"2d228d91-771d-42af-8750-ce9b823760ed\") " Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.631844 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data\") pod \"576113f5-04da-4871-980d-c101f55b1622\" (UID: \"576113f5-04da-4871-980d-c101f55b1622\") " Nov 02 17:52:30.632173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632040 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632818 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632845 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632861 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f070449e-b50d-405b-abeb-b251576ed8de-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632877 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632892 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632905 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdvs4\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-kube-api-access-kdvs4\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632919 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f070449e-b50d-405b-abeb-b251576ed8de-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.632991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.632933 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f070449e-b50d-405b-abeb-b251576ed8de-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.633626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.633107 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.633626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.633290 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.633626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.633394 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:30.633626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.633541 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.633626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.633591 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:30.633947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.633877 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.634206 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.634164 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.634427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.634394 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.634616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.634593 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.635066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.634650 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.635066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.634658 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.635066 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.634887 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:30.635397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.635354 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "kube-api-access-4cn9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.637863 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.637815 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.639940 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.639967 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.639991 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "kube-api-access-srzd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.640049 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.640079 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.640117 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy" pod "2d228d91-771d-42af-8750-ce9b823760ed" (UID: "2d228d91-771d-42af-8750-ce9b823760ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.640172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.640147 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.640609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.640243 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.641140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.641108 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy" pod "576113f5-04da-4871-980d-c101f55b1622" (UID: "576113f5-04da-4871-980d-c101f55b1622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:30.641303 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.641284 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh" pod "3375acde-3ce8-442a-9718-e7a6799e9722" (UID: "3375acde-3ce8-442a-9718-e7a6799e9722"). InnerVolumeSpecName "kube-api-access-cpvnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:30.734846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734808 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.734846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734846 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734861 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734874 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734891 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzd4\" (UniqueName: \"kubernetes.io/projected/576113f5-04da-4871-980d-c101f55b1622-kube-api-access-srzd4\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734906 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3375acde-3ce8-442a-9718-e7a6799e9722-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734920 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734933 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d228d91-771d-42af-8750-ce9b823760ed-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734946 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d228d91-771d-42af-8750-ce9b823760ed-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734960 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734973 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576113f5-04da-4871-980d-c101f55b1622-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734986 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576113f5-04da-4871-980d-c101f55b1622-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.734999 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735012 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735024 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735036 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cn9k\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-kube-api-access-4cn9k\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735066 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d228d91-771d-42af-8750-ce9b823760ed-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735081 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735097 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvnh\" (UniqueName: \"kubernetes.io/projected/3375acde-3ce8-442a-9718-e7a6799e9722-kube-api-access-cpvnh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735114 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735129 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3375acde-3ce8-442a-9718-e7a6799e9722-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735143 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d228d91-771d-42af-8750-ce9b823760ed-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735157 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3375acde-3ce8-442a-9718-e7a6799e9722-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:30.735471 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:30.735170 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576113f5-04da-4871-980d-c101f55b1622-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:31.047816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.047718 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5"] Nov 02 17:52:31.050639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.050610 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b0a756-5xwl5"] Nov 02 17:52:31.081976 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.081943 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d249cc40-f495-4126-8a18-6d77d8b827e8" path="/var/lib/kubelet/pods/d249cc40-f495-4126-8a18-6d77d8b827e8/volumes" Nov 02 17:52:31.094691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.094661 5647 generic.go:356] "Generic (PLEG): container finished" podID="3375acde-3ce8-442a-9718-e7a6799e9722" containerID="fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b" exitCode=0 Nov 02 17:52:31.094865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.094714 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" Nov 02 17:52:31.094865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.094769 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" event={"ID":"3375acde-3ce8-442a-9718-e7a6799e9722","Type":"ContainerDied","Data":"fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b"} Nov 02 17:52:31.094865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.094830 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv" event={"ID":"3375acde-3ce8-442a-9718-e7a6799e9722","Type":"ContainerDied","Data":"4a6fa4d1937a994d11ec771e7d1d82eac03255512a8d6f733b8fc599048265f1"} Nov 02 17:52:31.094865 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.094854 5647 scope.go:122] "RemoveContainer" containerID="fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b" Nov 02 17:52:31.096694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.096675 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" Nov 02 17:52:31.097055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.096706 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" event={"ID":"f070449e-b50d-405b-abeb-b251576ed8de","Type":"ContainerDied","Data":"39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17"} Nov 02 17:52:31.097145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.096624 5647 generic.go:356] "Generic (PLEG): container finished" podID="f070449e-b50d-405b-abeb-b251576ed8de" containerID="39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17" exitCode=0 Nov 02 17:52:31.097250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.097174 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd" event={"ID":"f070449e-b50d-405b-abeb-b251576ed8de","Type":"ContainerDied","Data":"bdfaf0a5a0f65849aa2090e4cc436ff13728386c8089cef1da4216c9c836354a"} Nov 02 17:52:31.099086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.099058 5647 generic.go:356] "Generic (PLEG): container finished" podID="576113f5-04da-4871-980d-c101f55b1622" containerID="45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b" exitCode=0 Nov 02 17:52:31.099231 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.099215 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" event={"ID":"576113f5-04da-4871-980d-c101f55b1622","Type":"ContainerDied","Data":"45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b"} Nov 02 17:52:31.099288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.099260 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" event={"ID":"576113f5-04da-4871-980d-c101f55b1622","Type":"ContainerDied","Data":"917bd80fb52c3860d34420e5474f4167ba67b9917823b75960efa270bd2dcd2c"} Nov 02 17:52:31.099344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.099331 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm" Nov 02 17:52:31.101667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.101645 5647 generic.go:356] "Generic (PLEG): container finished" podID="2d228d91-771d-42af-8750-ce9b823760ed" containerID="07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285" exitCode=0 Nov 02 17:52:31.101884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.101675 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" event={"ID":"2d228d91-771d-42af-8750-ce9b823760ed","Type":"ContainerDied","Data":"07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285"} Nov 02 17:52:31.101984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.101912 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" event={"ID":"2d228d91-771d-42af-8750-ce9b823760ed","Type":"ContainerDied","Data":"40c56cccb067a0de559ee337d3faa1c9740581cda3020b0107154459d10a90e8"} Nov 02 17:52:31.101984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.101756 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv" Nov 02 17:52:31.108779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.108760 5647 scope.go:122] "RemoveContainer" containerID="fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b" Nov 02 17:52:31.109056 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:31.109041 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b\": container with ID starting with fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b not found: ID does not exist" containerID="fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b" Nov 02 17:52:31.109099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.109064 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b"} err="failed to get container status \"fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b\": rpc error: code = NotFound desc = could not find container \"fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b\": container with ID starting with fbef552431442a76c1811aa124d52f80db177fcca3b1c809091171406552209b not found: ID does not exist" Nov 02 17:52:31.109099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.109080 5647 scope.go:122] "RemoveContainer" containerID="39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17" Nov 02 17:52:31.118576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.118557 5647 scope.go:122] "RemoveContainer" containerID="39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17" Nov 02 17:52:31.118920 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:31.118895 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17\": container with ID starting with 39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17 not found: ID does not exist" containerID="39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17" Nov 02 17:52:31.119020 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.118934 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17"} err="failed to get container status \"39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17\": rpc error: code = NotFound desc = could not find container \"39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17\": container with ID starting with 39cf52dd5e387daab6ac1f47c761910ec80971be738dc41f399400aefe165d17 not found: ID does not exist" Nov 02 17:52:31.119020 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.118960 5647 scope.go:122] "RemoveContainer" containerID="45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b" Nov 02 17:52:31.155192 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.155169 5647 scope.go:122] "RemoveContainer" containerID="fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c" Nov 02 17:52:31.165150 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.165128 5647 scope.go:122] "RemoveContainer" containerID="45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b" Nov 02 17:52:31.165454 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:31.165433 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b\": container with ID starting with 45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b not found: ID does not exist" containerID="45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b" Nov 02 17:52:31.165501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.165470 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b"} err="failed to get container status \"45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b\": rpc error: code = NotFound desc = could not find container \"45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b\": container with ID starting with 45e58b19680256ad2c1fe5e07a6a99bfd73e5672f28e3e862d1206bcafb8766b not found: ID does not exist" Nov 02 17:52:31.165501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.165489 5647 scope.go:122] "RemoveContainer" containerID="fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c" Nov 02 17:52:31.165789 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:31.165768 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c\": container with ID starting with fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c not found: ID does not exist" containerID="fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c" Nov 02 17:52:31.165857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.165795 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c"} err="failed to get container status \"fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c\": rpc error: code = NotFound desc = could not find container \"fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c\": container with ID starting with fd2526762623fcfea11e376ad8b98c8f13cb875954f2bfc2e4b0afadb6eb5e0c not found: ID does not exist" Nov 02 17:52:31.165857 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.165811 5647 scope.go:122] "RemoveContainer" containerID="07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285" Nov 02 17:52:31.174928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.174884 5647 scope.go:122] "RemoveContainer" containerID="07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285" Nov 02 17:52:31.175190 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:31.175173 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285\": container with ID starting with 07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285 not found: ID does not exist" containerID="07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285" Nov 02 17:52:31.175264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.175194 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285"} err="failed to get container status \"07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285\": rpc error: code = NotFound desc = could not find container \"07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285\": container with ID starting with 07c0ea503cff1a2a2f5478e73731216e58aa2eb9717d795a309b743d0c7ba285 not found: ID does not exist" Nov 02 17:52:31.265138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.265105 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd"] Nov 02 17:52:31.270469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.270442 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-m7fkd"] Nov 02 17:52:31.382658 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.382627 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv"] Nov 02 17:52:31.387851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.387810 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-28knv"] Nov 02 17:52:31.497394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.497144 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv"] Nov 02 17:52:31.501061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.501037 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-gknlv"] Nov 02 17:52:31.612896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.612800 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm"] Nov 02 17:52:31.616668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:31.616642 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8cbedb9f-openshift-default-898d548fc-mrnxm"] Nov 02 17:52:32.354176 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.349233 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6"] Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354674 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354705 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354726 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d2112269-cd3d-4659-8156-9380e128dbde" containerName="payload-pre-processing" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354786 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2112269-cd3d-4659-8156-9380e128dbde" containerName="payload-pre-processing" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354810 5647 cpu_manager.go:414] "containerMap: removing container" podUID="884cc241-6b33-4240-8f51-3ac590b01b39" containerName="payload-processing" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354819 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884cc241-6b33-4240-8f51-3ac590b01b39" containerName="payload-processing" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354848 5647 cpu_manager.go:414] "containerMap: removing container" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354856 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354874 5647 cpu_manager.go:414] "containerMap: removing container" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354883 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354899 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354907 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354927 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.354935 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355100 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355126 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f070449e-b50d-405b-abeb-b251576ed8de" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355135 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2112269-cd3d-4659-8156-9380e128dbde" containerName="payload-pre-processing" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355148 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="884cc241-6b33-4240-8f51-3ac590b01b39" containerName="payload-processing" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355161 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355172 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="576113f5-04da-4871-980d-c101f55b1622" containerName="istio-proxy" Nov 02 17:52:32.355816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.355183 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d228d91-771d-42af-8750-ce9b823760ed" containerName="istio-proxy" Nov 02 17:52:32.376706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.376670 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.411014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.410976 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6"] Nov 02 17:52:32.430797 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.430727 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-666de0-dockercfg-lxqpt\"" Nov 02 17:52:32.446104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.446045 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmpvb\" (UniqueName: \"kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb\") pod \"payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.446304 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.446157 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.458242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.458211 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-666de0\"" Nov 02 17:52:32.547032 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.546991 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.547458 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.547165 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmpvb\" (UniqueName: \"kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb\") pod \"payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.547679 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.547660 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.608968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.608886 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmpvb\" (UniqueName: \"kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb\") pod \"payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:32.696665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:32.696630 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:33.078345 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:52:33.078313 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1b15a3_a48f_4722_9976_66b8a0472947.slice/crio-b2e301da3dc8b754cd94744d42936b7c2d5c47aa5ce7c031b17ca19f2f74aff5 WatchSource:0}: Error finding container b2e301da3dc8b754cd94744d42936b7c2d5c47aa5ce7c031b17ca19f2f74aff5: Status 404 returned error can't find the container with id b2e301da3dc8b754cd94744d42936b7c2d5c47aa5ce7c031b17ca19f2f74aff5 Nov 02 17:52:33.083770 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:33.083718 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d228d91-771d-42af-8750-ce9b823760ed" path="/var/lib/kubelet/pods/2d228d91-771d-42af-8750-ce9b823760ed/volumes" Nov 02 17:52:33.084308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:33.084288 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3375acde-3ce8-442a-9718-e7a6799e9722" path="/var/lib/kubelet/pods/3375acde-3ce8-442a-9718-e7a6799e9722/volumes" Nov 02 17:52:33.084806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:33.084787 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576113f5-04da-4871-980d-c101f55b1622" path="/var/lib/kubelet/pods/576113f5-04da-4871-980d-c101f55b1622/volumes" Nov 02 17:52:33.085579 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:33.085557 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f070449e-b50d-405b-abeb-b251576ed8de" path="/var/lib/kubelet/pods/f070449e-b50d-405b-abeb-b251576ed8de/volumes" Nov 02 17:52:33.085952 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:33.085934 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6"] Nov 02 17:52:33.158778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:33.158728 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" event={"ID":"ec1b15a3-a48f-4722-9976-66b8a0472947","Type":"ContainerStarted","Data":"b2e301da3dc8b754cd94744d42936b7c2d5c47aa5ce7c031b17ca19f2f74aff5"} Nov 02 17:52:34.168650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.168601 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" event={"ID":"ec1b15a3-a48f-4722-9976-66b8a0472947","Type":"ContainerStarted","Data":"6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b"} Nov 02 17:52:34.169118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.168758 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:34.288337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.288268 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" podStartSLOduration=2.288247667 podStartE2EDuration="2.288247667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:34.284181375 +0000 UTC m=+1741.357708580" watchObservedRunningTime="2026-11-02 17:52:34.288247667 +0000 UTC m=+1741.361774871" Nov 02 17:52:34.973669 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.973638 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd"] Nov 02 17:52:34.977833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.977816 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:34.992145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.992119 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:34.992270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.992173 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:34.992337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.992318 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:34.992435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:34.992422 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7b8g\" (UniqueName: \"kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.033190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.033095 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-700abe-serving-cert\"" Nov 02 17:52:35.036806 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.036779 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd"] Nov 02 17:52:35.093618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.093581 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.093822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.093642 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.093822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.093708 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.093822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.093795 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7b8g\" (UniqueName: \"kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.094240 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:35.093848 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-serving-cert: secret "maas-api-e2e-ipp-a-700abe-serving-cert" not found Nov 02 17:52:35.094240 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:35.093978 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls podName:94fa3f47-4c8c-4fb6-a903-85502bb1ff4c nodeName:}" failed. No retries permitted until 2026-11-02 17:52:35.593955988 +0000 UTC m=+1742.667483169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls") pod "maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" (UID: "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c") : secret "maas-api-e2e-ipp-a-700abe-serving-cert" not found Nov 02 17:52:35.094482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.094425 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.096699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.096667 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.153446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.153411 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7b8g\" (UniqueName: \"kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.607231 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.607198 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.609760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.609720 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.686234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.686189 5647 scope.go:122] "RemoveContainer" containerID="07fb9680b27b8abc422938cfac6cb55f9886fb1e2582b746d9973dff29dc1abd" Nov 02 17:52:35.892121 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:35.892022 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:35.958619 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:35.958579 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:52:36.279769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:36.279723 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd"] Nov 02 17:52:36.281989 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:52:36.281952 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fa3f47_4c8c_4fb6_a903_85502bb1ff4c.slice/crio-f40d12d533d0b24fb90d0c6c6965be80391cdf64e4512d88ba7c378a6e112441 WatchSource:0}: Error finding container f40d12d533d0b24fb90d0c6c6965be80391cdf64e4512d88ba7c378a6e112441: Status 404 returned error can't find the container with id f40d12d533d0b24fb90d0c6c6965be80391cdf64e4512d88ba7c378a6e112441 Nov 02 17:52:36.283832 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:36.283807 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:52:37.192000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:37.191966 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" event={"ID":"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c","Type":"ContainerStarted","Data":"f40d12d533d0b24fb90d0c6c6965be80391cdf64e4512d88ba7c378a6e112441"} Nov 02 17:52:38.202990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:38.202947 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" event={"ID":"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c","Type":"ContainerStarted","Data":"15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521"} Nov 02 17:52:38.203863 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:38.203836 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:38.320891 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:38.320841 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" podStartSLOduration=3.405737145 podStartE2EDuration="4.320827557s" totalImagesPullingTime="915.090412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:38.320355294 +0000 UTC m=+1745.393882501" watchObservedRunningTime="2026-11-02 17:52:38.320827557 +0000 UTC m=+1745.394354760" Nov 02 17:52:39.619953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.603200 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv"] Nov 02 17:52:39.633178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.633150 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv"] Nov 02 17:52:39.633540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.633518 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.688882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.688837 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfcn\" (UniqueName: \"kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn\") pod \"payload-processing-e2e-api-a-5bf666-69465559c-bsfgv\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.689087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.688917 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5bf666-69465559c-bsfgv\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.714397 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.714363 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5bf666\"" Nov 02 17:52:39.714612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.714418 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5bf666-dockercfg-7n5fg\"" Nov 02 17:52:39.779094 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.779055 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:52:39.812761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.808917 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfcn\" (UniqueName: \"kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn\") pod \"payload-processing-e2e-api-a-5bf666-69465559c-bsfgv\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.812761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.809002 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5bf666-69465559c-bsfgv\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.812761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.810237 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5bf666-69465559c-bsfgv\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.890144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.890054 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfcn\" (UniqueName: \"kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn\") pod \"payload-processing-e2e-api-a-5bf666-69465559c-bsfgv\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:39.996353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:39.995060 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:40.489789 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:40.489758 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv"] Nov 02 17:52:40.491361 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:52:40.491331 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2f65b6_8e4f_419b_87a3_ab40983e8ec6.slice/crio-d3779b07ff2d79f4c6487c6d3b5385d32bcc5f2b81c9b171e99bff812efdc09d WatchSource:0}: Error finding container d3779b07ff2d79f4c6487c6d3b5385d32bcc5f2b81c9b171e99bff812efdc09d: Status 404 returned error can't find the container with id d3779b07ff2d79f4c6487c6d3b5385d32bcc5f2b81c9b171e99bff812efdc09d Nov 02 17:52:41.242776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:41.241160 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" event={"ID":"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6","Type":"ContainerStarted","Data":"dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd"} Nov 02 17:52:41.242776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:41.241257 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" event={"ID":"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6","Type":"ContainerStarted","Data":"d3779b07ff2d79f4c6487c6d3b5385d32bcc5f2b81c9b171e99bff812efdc09d"} Nov 02 17:52:41.242776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:41.242234 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:41.359086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:41.359018 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" podStartSLOduration=2.359000693 podStartE2EDuration="2.359000693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:41.357794812 +0000 UTC m=+1748.431322016" watchObservedRunningTime="2026-11-02 17:52:41.359000693 +0000 UTC m=+1748.432527900" Nov 02 17:52:44.214861 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:44.214829 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:52:45.178897 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:45.178866 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:52:47.725379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:47.725345 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz"] Nov 02 17:52:47.725896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:47.725580 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" containerID="cri-o://1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7" gracePeriod=30 Nov 02 17:52:52.251726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:52.251693 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:52:53.008033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.007976 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:52:53.129779 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.129729 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.129963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.129793 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.129963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.129899 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.129963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.129952 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.130105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.129976 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.130105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130013 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.130105 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130047 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.130276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130119 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.130276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130212 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.130276 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130228 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert\") pod \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\" (UID: \"bfaf2ab1-8a7a-45cb-8741-f8e266a87751\") " Nov 02 17:52:53.130427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130268 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.130427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130310 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.130525 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130487 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.130569 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.130523 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:52:53.131586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.131486 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.131586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.131513 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.131586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.131531 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.131586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.131547 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.131586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.131562 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.132522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.132495 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "kube-api-access-s6srw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:53.132681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.132654 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:52:53.132844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.132726 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:52:53.132844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.132830 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy" pod "bfaf2ab1-8a7a-45cb-8741-f8e266a87751" (UID: "bfaf2ab1-8a7a-45cb-8741-f8e266a87751"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:52:53.232990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.232955 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.232990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.232983 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.232990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.232993 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6srw\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-kube-api-access-s6srw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.233248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.233004 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfaf2ab1-8a7a-45cb-8741-f8e266a87751-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:52:53.376213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.376176 5647 generic.go:356] "Generic (PLEG): container finished" podID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerID="1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7" exitCode=0 Nov 02 17:52:53.376638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.376236 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" Nov 02 17:52:53.376638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.376250 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" event={"ID":"bfaf2ab1-8a7a-45cb-8741-f8e266a87751","Type":"ContainerDied","Data":"1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7"} Nov 02 17:52:53.376638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.376310 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz" event={"ID":"bfaf2ab1-8a7a-45cb-8741-f8e266a87751","Type":"ContainerDied","Data":"2c8f9d0540f5dc44c000f034be65c05d2cebb96064ccde76f6696522538cbdcb"} Nov 02 17:52:53.376638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.376335 5647 scope.go:122] "RemoveContainer" containerID="1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7" Nov 02 17:52:53.387556 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.387537 5647 scope.go:122] "RemoveContainer" containerID="44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c" Nov 02 17:52:53.397220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.397199 5647 scope.go:122] "RemoveContainer" containerID="1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7" Nov 02 17:52:53.397533 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:53.397506 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7\": container with ID starting with 1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7 not found: ID does not exist" containerID="1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7" Nov 02 17:52:53.397635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.397541 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7"} err="failed to get container status \"1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7\": rpc error: code = NotFound desc = could not find container \"1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7\": container with ID starting with 1ba48b88d17d89644f456600240130572f33ea1d7841875ef23c2011450751d7 not found: ID does not exist" Nov 02 17:52:53.397635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.397568 5647 scope.go:122] "RemoveContainer" containerID="44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c" Nov 02 17:52:53.397863 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:52:53.397844 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c\": container with ID starting with 44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c not found: ID does not exist" containerID="44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c" Nov 02 17:52:53.397921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.397869 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c"} err="failed to get container status \"44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c\": rpc error: code = NotFound desc = could not find container \"44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c\": container with ID starting with 44b0459b986c34ff705a3c7b56c90d8a9e4849a1704ae74e79338cead28b6d5c not found: ID does not exist" Nov 02 17:52:53.545541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.545507 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz"] Nov 02 17:52:53.551437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:53.551406 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e61288-openshift-default-5d47f4bcb9-vbggz"] Nov 02 17:52:55.081459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:55.081424 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" path="/var/lib/kubelet/pods/bfaf2ab1-8a7a-45cb-8741-f8e266a87751/volumes" Nov 02 17:52:55.202024 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:55.201933 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn"] Nov 02 17:52:55.320874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:52:55.320227 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw"] Nov 02 17:53:00.202844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.202777 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" podUID="e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" containerName="payload-processing" containerID="cri-o://410dc93745329c90ac847124eceece0154ffbd50048784e0c2ada4574fdb5452" gracePeriod=25 Nov 02 17:53:00.321271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.321199 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" podUID="ff3d3f6e-9e53-43d9-9194-0cc639023629" containerName="payload-pre-processing" containerID="cri-o://b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d" gracePeriod=25 Nov 02 17:53:00.458083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.457804 5647 generic.go:356] "Generic (PLEG): container finished" podID="e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" containerID="410dc93745329c90ac847124eceece0154ffbd50048784e0c2ada4574fdb5452" exitCode=0 Nov 02 17:53:00.458083 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.457967 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" event={"ID":"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a","Type":"ContainerDied","Data":"410dc93745329c90ac847124eceece0154ffbd50048784e0c2ada4574fdb5452"} Nov 02 17:53:00.537243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.537211 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:53:00.670259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.670234 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:53:00.686500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.686464 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume\") pod \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " Nov 02 17:53:00.686790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.686681 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87\" (UniqueName: \"kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87\") pod \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\" (UID: \"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a\") " Nov 02 17:53:00.687308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.687281 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume" pod "e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" (UID: "e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:00.689170 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.689149 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87" pod "e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" (UID: "e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a"). InnerVolumeSpecName "kube-api-access-xpt87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:00.788280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.788188 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz\" (UniqueName: \"kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz\") pod \"ff3d3f6e-9e53-43d9-9194-0cc639023629\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " Nov 02 17:53:00.788463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.788309 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume\") pod \"ff3d3f6e-9e53-43d9-9194-0cc639023629\" (UID: \"ff3d3f6e-9e53-43d9-9194-0cc639023629\") " Nov 02 17:53:00.789183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.788891 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:00.789183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.788917 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpt87\" (UniqueName: \"kubernetes.io/projected/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a-kube-api-access-xpt87\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:00.789440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.789193 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume" pod "ff3d3f6e-9e53-43d9-9194-0cc639023629" (UID: "ff3d3f6e-9e53-43d9-9194-0cc639023629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:00.791350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.791322 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz" pod "ff3d3f6e-9e53-43d9-9194-0cc639023629" (UID: "ff3d3f6e-9e53-43d9-9194-0cc639023629"). InnerVolumeSpecName "kube-api-access-9fdbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:00.890418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.890383 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff3d3f6e-9e53-43d9-9194-0cc639023629-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:00.890418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:00.890414 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fdbz\" (UniqueName: \"kubernetes.io/projected/ff3d3f6e-9e53-43d9-9194-0cc639023629-kube-api-access-9fdbz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:01.328081 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.328032 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk"] Nov 02 17:53:01.329034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329007 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ff3d3f6e-9e53-43d9-9194-0cc639023629" containerName="payload-pre-processing" Nov 02 17:53:01.329202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329185 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3d3f6e-9e53-43d9-9194-0cc639023629" containerName="payload-pre-processing" Nov 02 17:53:01.329353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329340 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" Nov 02 17:53:01.329464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329452 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" Nov 02 17:53:01.329554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329544 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" Nov 02 17:53:01.329629 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329619 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" Nov 02 17:53:01.329719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329709 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" containerName="payload-processing" Nov 02 17:53:01.329836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.329823 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" containerName="payload-processing" Nov 02 17:53:01.330112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.330095 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3d3f6e-9e53-43d9-9194-0cc639023629" containerName="payload-pre-processing" Nov 02 17:53:01.330237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.330226 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" Nov 02 17:53:01.330321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.330309 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfaf2ab1-8a7a-45cb-8741-f8e266a87751" containerName="istio-proxy" Nov 02 17:53:01.330408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.330396 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" containerName="payload-processing" Nov 02 17:53:01.334638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.334611 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.439062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.438971 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f10385-serving-cert\"" Nov 02 17:53:01.442033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.442000 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk"] Nov 02 17:53:01.466987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.466956 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" Nov 02 17:53:01.466987 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.466979 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn" event={"ID":"e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a","Type":"ContainerDied","Data":"bf36f48c7c531e30b18c2e056bb654a08af330f5098fb3a07f67d949df4a09f7"} Nov 02 17:53:01.467238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.467041 5647 scope.go:122] "RemoveContainer" containerID="410dc93745329c90ac847124eceece0154ffbd50048784e0c2ada4574fdb5452" Nov 02 17:53:01.469026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.468536 5647 generic.go:356] "Generic (PLEG): container finished" podID="ff3d3f6e-9e53-43d9-9194-0cc639023629" containerID="b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d" exitCode=0 Nov 02 17:53:01.469026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.468602 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" Nov 02 17:53:01.469026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.468622 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" event={"ID":"ff3d3f6e-9e53-43d9-9194-0cc639023629","Type":"ContainerDied","Data":"b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d"} Nov 02 17:53:01.469227 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.469025 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw" event={"ID":"ff3d3f6e-9e53-43d9-9194-0cc639023629","Type":"ContainerDied","Data":"48f5f4e04a3b66af7ae4401f51c55539a8ba4bab674c7a33d9316bebefb45db4"} Nov 02 17:53:01.478458 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.478434 5647 scope.go:122] "RemoveContainer" containerID="b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d" Nov 02 17:53:01.488973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.488944 5647 scope.go:122] "RemoveContainer" containerID="b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d" Nov 02 17:53:01.489363 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:53:01.489340 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d\": container with ID starting with b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d not found: ID does not exist" containerID="b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d" Nov 02 17:53:01.489469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.489372 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d"} err="failed to get container status \"b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d\": rpc error: code = NotFound desc = could not find container \"b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d\": container with ID starting with b89fac8d1f46d4e96a0a7ef0f38d6aaa58a1b396248edc50cadf28ddefa57f6d not found: ID does not exist" Nov 02 17:53:01.499671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.499633 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.500089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.499970 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.500089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.500042 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.500321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.500302 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmcss\" (UniqueName: \"kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.611846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.610712 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.611846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.610859 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.611846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.610905 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.611846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.611034 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmcss\" (UniqueName: \"kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.612398 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:53:01.612375 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-serving-cert: secret "maas-api-e2e-ipp-b-f10385-serving-cert" not found Nov 02 17:53:01.619776 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:53:01.612573 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls podName:a3f8bb41-341d-47bd-96c2-d61e3710434f nodeName:}" failed. No retries permitted until 2026-11-02 17:53:02.112548232 +0000 UTC m=+1769.186075419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls") pod "maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" (UID: "a3f8bb41-341d-47bd-96c2-d61e3710434f") : secret "maas-api-e2e-ipp-b-f10385-serving-cert" not found Nov 02 17:53:01.620030 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.619612 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.659759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.657790 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn"] Nov 02 17:53:01.681614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.680979 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-27ab38-66f9bd7fc5-h68jn"] Nov 02 17:53:01.690242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.689848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmcss\" (UniqueName: \"kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.692795 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.692709 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:01.737265 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.737226 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw"] Nov 02 17:53:01.740097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:01.740066 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-27ab38-64c66699bc-bcghw"] Nov 02 17:53:02.129811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:02.129765 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:02.132484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:02.132453 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:02.249531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:02.249491 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:02.656594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:02.656557 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk"] Nov 02 17:53:02.657427 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:53:02.657382 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f8bb41_341d_47bd_96c2_d61e3710434f.slice/crio-da4511a00747a96266ce56740f7d54f2db974187dc1994febd6e873519c92a62 WatchSource:0}: Error finding container da4511a00747a96266ce56740f7d54f2db974187dc1994febd6e873519c92a62: Status 404 returned error can't find the container with id da4511a00747a96266ce56740f7d54f2db974187dc1994febd6e873519c92a62 Nov 02 17:53:03.081655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:03.081578 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a" path="/var/lib/kubelet/pods/e68ce3d4-faf6-49c2-aa4e-b36b3fe1e50a/volumes" Nov 02 17:53:03.082043 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:03.082028 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3d3f6e-9e53-43d9-9194-0cc639023629" path="/var/lib/kubelet/pods/ff3d3f6e-9e53-43d9-9194-0cc639023629/volumes" Nov 02 17:53:03.486563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:03.486525 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" event={"ID":"a3f8bb41-341d-47bd-96c2-d61e3710434f","Type":"ContainerStarted","Data":"da4511a00747a96266ce56740f7d54f2db974187dc1994febd6e873519c92a62"} Nov 02 17:53:04.496754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:04.494612 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" event={"ID":"a3f8bb41-341d-47bd-96c2-d61e3710434f","Type":"ContainerStarted","Data":"73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd"} Nov 02 17:53:04.496754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:04.494692 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:05.949581 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:53:05.949543 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:53:08.995949 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:08.995897 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:08.996643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:08.996601 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:10.509163 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.509132 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:53:10.540230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.540195 5647 generic.go:356] "Generic (PLEG): container finished" podID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerID="4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f" exitCode=0 Nov 02 17:53:10.540398 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.540243 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" event={"ID":"9e00115b-49bd-4a20-b83b-c845f2b06343","Type":"ContainerDied","Data":"4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f"} Nov 02 17:53:10.540699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.540684 5647 scope.go:122] "RemoveContainer" containerID="4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f" Nov 02 17:53:10.542316 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.542282 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:10.542429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.542361 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:10.542429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.542401 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:10.628684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:10.628632 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" podStartSLOduration=8.748989357 podStartE2EDuration="9.628615001s" totalImagesPullingTime="879.625644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:04.628014978 +0000 UTC m=+1771.701542184" watchObservedRunningTime="2026-11-02 17:53:10.628615001 +0000 UTC m=+1777.702142204" Nov 02 17:53:11.054757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:11.051129 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv"] Nov 02 17:53:11.057959 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:11.056058 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125fbf55-5hgsv"] Nov 02 17:53:11.081522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:11.081485 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4dffac-203a-422d-a4f8-a3a75ced37e3" path="/var/lib/kubelet/pods/6e4dffac-203a-422d-a4f8-a3a75ced37e3/volumes" Nov 02 17:53:11.548212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:11.548173 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" event={"ID":"9e00115b-49bd-4a20-b83b-c845f2b06343","Type":"ContainerStarted","Data":"0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b"} Nov 02 17:53:12.326799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:12.323263 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:53:12.326799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:12.323444 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 17:53:12.326799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:12.323502 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 17:53:13.325761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:13.322119 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 17:53:13.325761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:13.322202 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 17:53:14.327229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:14.327186 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 17:53:14.327652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:14.327278 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 17:53:15.326762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:15.326389 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 17:53:15.326762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:15.326474 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 17:53:16.327199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.327100 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Nov 02 17:53:16.327199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.327183 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Nov 02 17:53:16.845469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.845431 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k"] Nov 02 17:53:16.849240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.849220 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:16.873688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.873657 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:16.873872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.873717 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grfq\" (UniqueName: \"kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:16.905520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.905489 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k"] Nov 02 17:53:16.953265 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.953237 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6f23bd\"" Nov 02 17:53:16.953510 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.953388 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6f23bd-dockercfg-hjgwx\"" Nov 02 17:53:16.975302 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.975273 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grfq\" (UniqueName: \"kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:16.975558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.975539 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:16.976317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:16.976290 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:17.033381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.033340 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grfq\" (UniqueName: \"kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq\") pod \"payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:17.163006 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.162926 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:17.326379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.326349 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:53:17.326878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.326857 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:53:17.327576 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.327560 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:53:17.544319 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.544236 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv"] Nov 02 17:53:17.547501 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.547480 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.565036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.565015 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k"] Nov 02 17:53:17.587841 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.587799 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.588181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.588159 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wn6h\" (UniqueName: \"kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h\") pod \"payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.609014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.608934 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv"] Nov 02 17:53:17.626343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.626301 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" event={"ID":"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27","Type":"ContainerStarted","Data":"9917aae436770ee8777e22c3ab04f57f87143f8b8605fb98efe344d7805d873d"} Nov 02 17:53:17.691096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.691056 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.691269 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.691169 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wn6h\" (UniqueName: \"kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h\") pod \"payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.692154 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.692128 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.749051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.749017 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wn6h\" (UniqueName: \"kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h\") pod \"payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:17.864014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:17.863981 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:18.241538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.241501 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv"] Nov 02 17:53:18.242396 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:53:18.242371 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf189dc21_699c_4947_84c6_200e7f4adc4a.slice/crio-846e443c6eab83eba0468e5d3da9dbbca4996102d9af82be3ae9207243231340 WatchSource:0}: Error finding container 846e443c6eab83eba0468e5d3da9dbbca4996102d9af82be3ae9207243231340: Status 404 returned error can't find the container with id 846e443c6eab83eba0468e5d3da9dbbca4996102d9af82be3ae9207243231340 Nov 02 17:53:18.641462 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.641422 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" event={"ID":"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27","Type":"ContainerStarted","Data":"47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e"} Nov 02 17:53:18.642190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.642168 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:18.652356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.652208 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" event={"ID":"f189dc21-699c-4947-84c6-200e7f4adc4a","Type":"ContainerStarted","Data":"54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392"} Nov 02 17:53:18.652356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.652269 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" event={"ID":"f189dc21-699c-4947-84c6-200e7f4adc4a","Type":"ContainerStarted","Data":"846e443c6eab83eba0468e5d3da9dbbca4996102d9af82be3ae9207243231340"} Nov 02 17:53:18.652356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.652298 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:18.759360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:18.759307 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" podStartSLOduration=2.759293613 podStartE2EDuration="2.759293613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:18.757437101 +0000 UTC m=+1785.830964316" watchObservedRunningTime="2026-11-02 17:53:18.759293613 +0000 UTC m=+1785.832820807" Nov 02 17:53:23.483782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:23.477433 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" podStartSLOduration=6.47741479 podStartE2EDuration="6.47741479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:18.904887134 +0000 UTC m=+1785.978414338" watchObservedRunningTime="2026-11-02 17:53:23.47741479 +0000 UTC m=+1790.550941996" Nov 02 17:53:23.510762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:23.505783 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:53:27.987486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:27.987451 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk"] Nov 02 17:53:27.991864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:27.991845 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.095921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.095885 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-666de0-serving-cert\"" Nov 02 17:53:28.098793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.098741 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw"] Nov 02 17:53:28.101465 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.101444 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk"] Nov 02 17:53:28.123202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.101705 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8pp\" (UniqueName: \"kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.123202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.101851 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.123202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.101885 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.123202 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.101984 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.133435 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.133401 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f56f9ac-2jhkw"] Nov 02 17:53:28.225761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.225644 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8pp\" (UniqueName: \"kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.225761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.225751 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.227828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.226523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.227828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.226644 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.228595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.228567 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.229480 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.229455 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.230428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.230405 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.285086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.285011 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8pp\" (UniqueName: \"kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp\") pod \"maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.328850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.328817 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:28.730691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.730658 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk"] Nov 02 17:53:28.731896 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:53:28.731867 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod010bf490_d048_49d4_b7c9_ac2dcd50c4f3.slice/crio-c4fc7e934b9bcbae2527f6685dab91ab3b46cbdc85478c24dcbc0b3dd3cda357 WatchSource:0}: Error finding container c4fc7e934b9bcbae2527f6685dab91ab3b46cbdc85478c24dcbc0b3dd3cda357: Status 404 returned error can't find the container with id c4fc7e934b9bcbae2527f6685dab91ab3b46cbdc85478c24dcbc0b3dd3cda357 Nov 02 17:53:28.743144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:28.743119 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" event={"ID":"010bf490-d048-49d4-b7c9-ac2dcd50c4f3","Type":"ContainerStarted","Data":"c4fc7e934b9bcbae2527f6685dab91ab3b46cbdc85478c24dcbc0b3dd3cda357"} Nov 02 17:53:29.081714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:29.081625 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="064e172b-a6a9-466c-a233-9ccd20ede71c" path="/var/lib/kubelet/pods/064e172b-a6a9-466c-a233-9ccd20ede71c/volumes" Nov 02 17:53:29.661065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:29.661040 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:53:29.661234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:29.661110 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:53:29.761654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:29.761620 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" event={"ID":"010bf490-d048-49d4-b7c9-ac2dcd50c4f3","Type":"ContainerStarted","Data":"5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9"} Nov 02 17:53:29.761852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:29.761722 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:30.035784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:30.035692 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" podStartSLOduration=2.194237219 podStartE2EDuration="3.035674247s" totalImagesPullingTime="841.437028ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:30.035184815 +0000 UTC m=+1797.108712020" watchObservedRunningTime="2026-11-02 17:53:30.035674247 +0000 UTC m=+1797.109201450" Nov 02 17:53:33.462881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:33.462833 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 17:53:33.463291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:33.462888 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 17:53:34.123460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.123431 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:53:34.136340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.136312 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:53:34.807199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.807161 5647 generic.go:356] "Generic (PLEG): container finished" podID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerID="b940f52dd8abe7e55bc5453b4d82498a2165bbb72f2226655e5df951723273b5" exitCode=0 Nov 02 17:53:34.807646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.807233 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" event={"ID":"d01ea49e-000a-40ec-b40c-d461cc102fbf","Type":"ContainerDied","Data":"b940f52dd8abe7e55bc5453b4d82498a2165bbb72f2226655e5df951723273b5"} Nov 02 17:53:34.807696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.807678 5647 scope.go:122] "RemoveContainer" containerID="b940f52dd8abe7e55bc5453b4d82498a2165bbb72f2226655e5df951723273b5" Nov 02 17:53:34.809414 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.809384 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:34.809491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.809453 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:34.809491 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:34.809485 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:35.764064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:35.764031 5647 scope.go:122] "RemoveContainer" containerID="796e7e977f680bbf791c714652a8e5fb4cb9de0f53075e87ab49ca1db619574f" Nov 02 17:53:35.773695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:35.773672 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:53:35.835074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:35.835039 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" event={"ID":"d01ea49e-000a-40ec-b40c-d461cc102fbf","Type":"ContainerStarted","Data":"666b959e6e7d97bb35e05381e1f9ed68051f2e2397ae625bb78dda06169883ab"} Nov 02 17:53:35.970214 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:53:35.970171 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:53:36.468418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:36.468383 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:53:36.468662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:36.468588 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:53:36.468662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:36.468639 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 17:53:37.468459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:37.468420 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:53:37.468864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:37.468480 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 17:53:38.469557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:38.469519 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:53:38.470000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:38.469608 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 17:53:39.468767 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:39.468713 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:53:39.468964 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:39.468806 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 17:53:40.468710 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:40.468673 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 02 17:53:40.469167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:40.468775 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 02 17:53:41.472984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:41.472905 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:53:41.473384 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:41.473178 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:53:41.473829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:41.473812 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:53:52.995641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:52.995603 5647 generic.go:356] "Generic (PLEG): container finished" podID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerID="109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115" exitCode=0 Nov 02 17:53:52.996163 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:52.995675 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" event={"ID":"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9","Type":"ContainerDied","Data":"109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115"} Nov 02 17:53:52.996214 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:52.996191 5647 scope.go:122] "RemoveContainer" containerID="109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115" Nov 02 17:53:52.997971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:52.997930 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:52.998090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:52.998022 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:52.998090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:52.998068 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:53:54.166673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:54.166427 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" event={"ID":"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9","Type":"ContainerStarted","Data":"56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f"} Nov 02 17:53:54.933015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:54.932964 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:53:54.933482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:54.933459 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 17:53:54.933706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:54.933670 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 17:53:55.896350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:55.896313 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 17:53:55.896835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:55.896379 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 17:53:56.029183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:56.029136 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:53:56.895595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:56.895560 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 17:53:56.895803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:56.895611 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 17:53:57.548888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.548848 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw"] Nov 02 17:53:57.549311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.549062 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" containerID="cri-o://7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3" gracePeriod=30 Nov 02 17:53:57.636905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.636861 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs"] Nov 02 17:53:57.637240 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.637207 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" containerID="cri-o://934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300" gracePeriod=30 Nov 02 17:53:57.642764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.642715 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk"] Nov 02 17:53:57.643243 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.643213 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" containerID="cri-o://f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3" gracePeriod=30 Nov 02 17:53:57.896425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.896390 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 17:53:57.896614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:57.896447 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 17:53:58.895661 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:58.895614 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 02 17:53:58.896146 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:58.895678 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 02 17:53:59.036803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.036762 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l"] Nov 02 17:53:59.037048 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.037021 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" podUID="cb758a75-cf10-4257-a29e-c3f05157d38d" containerName="maas-api" containerID="cri-o://9abc5820d83119ec202f6e0b4c48d2d4506c5b2653ed6b4422441a45f61beaa5" gracePeriod=30 Nov 02 17:53:59.301253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.301184 5647 generic.go:356] "Generic (PLEG): container finished" podID="cb758a75-cf10-4257-a29e-c3f05157d38d" containerID="9abc5820d83119ec202f6e0b4c48d2d4506c5b2653ed6b4422441a45f61beaa5" exitCode=0 Nov 02 17:53:59.301784 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.301725 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" event={"ID":"cb758a75-cf10-4257-a29e-c3f05157d38d","Type":"ContainerDied","Data":"9abc5820d83119ec202f6e0b4c48d2d4506c5b2653ed6b4422441a45f61beaa5"} Nov 02 17:53:59.426091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.423625 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:53:59.578472 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.578213 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\") pod \"cb758a75-cf10-4257-a29e-c3f05157d38d\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " Nov 02 17:53:59.578472 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.578359 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf\" (UniqueName: \"kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf\") pod \"cb758a75-cf10-4257-a29e-c3f05157d38d\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " Nov 02 17:53:59.578756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.578507 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs\") pod \"cb758a75-cf10-4257-a29e-c3f05157d38d\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " Nov 02 17:53:59.578756 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.578564 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca\") pod \"cb758a75-cf10-4257-a29e-c3f05157d38d\" (UID: \"cb758a75-cf10-4257-a29e-c3f05157d38d\") " Nov 02 17:53:59.583484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.581949 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca" pod "cb758a75-cf10-4257-a29e-c3f05157d38d" (UID: "cb758a75-cf10-4257-a29e-c3f05157d38d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:53:59.603651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.603610 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls" pod "cb758a75-cf10-4257-a29e-c3f05157d38d" (UID: "cb758a75-cf10-4257-a29e-c3f05157d38d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:59.603898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.603746 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs" pod "cb758a75-cf10-4257-a29e-c3f05157d38d" (UID: "cb758a75-cf10-4257-a29e-c3f05157d38d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:53:59.606663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.606638 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf" pod "cb758a75-cf10-4257-a29e-c3f05157d38d" (UID: "cb758a75-cf10-4257-a29e-c3f05157d38d"). InnerVolumeSpecName "kube-api-access-n5vpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:53:59.679921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.679840 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vpf\" (UniqueName: \"kubernetes.io/projected/cb758a75-cf10-4257-a29e-c3f05157d38d-kube-api-access-n5vpf\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:59.679921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.679871 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:59.679921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.679881 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb758a75-cf10-4257-a29e-c3f05157d38d-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:59.679921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.679889 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb758a75-cf10-4257-a29e-c3f05157d38d-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:53:59.900650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.900619 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:53:59.901598 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:53:59.901577 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:54:00.310484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:00.310453 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" Nov 02 17:54:00.310484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:00.310476 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l" event={"ID":"cb758a75-cf10-4257-a29e-c3f05157d38d","Type":"ContainerDied","Data":"9a42142fdb513f4803995721d989e1942add76f65ade591e5a0b3d87c69b369e"} Nov 02 17:54:00.310790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:00.310534 5647 scope.go:122] "RemoveContainer" containerID="9abc5820d83119ec202f6e0b4c48d2d4506c5b2653ed6b4422441a45f61beaa5" Nov 02 17:54:00.479648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:00.479612 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l"] Nov 02 17:54:00.482296 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:00.482273 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-27ab38-848dcc546d-zn48l"] Nov 02 17:54:01.082578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:01.082542 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb758a75-cf10-4257-a29e-c3f05157d38d" path="/var/lib/kubelet/pods/cb758a75-cf10-4257-a29e-c3f05157d38d/volumes" Nov 02 17:54:01.417091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:01.417048 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:54:01.417278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:01.417116 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:54:02.835074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.834671 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:54:02.915995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.915956 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:54:02.936505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936470 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.936505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936509 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.936790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936556 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.936790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936597 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.936790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936630 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.936790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936657 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936803 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.937010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936872 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.937010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936878 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937010 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.936981 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937045 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.937248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937074 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.937248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937119 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.937248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937195 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.937248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937232 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937318 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937348 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937517 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937425 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs\") pod \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\" (UID: \"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9\") " Nov 02 17:54:02.937691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937577 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket\") pod \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\" (UID: \"c9de4197-e8e0-4abd-a991-d3d9e89e4b45\") " Nov 02 17:54:02.937771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.937726 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:02.938075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938046 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.938291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938268 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.938369 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938270 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.938868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938838 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:02.939004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938872 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:02.939004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938890 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:02.939004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938906 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:02.939004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.938922 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:02.939239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939173 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939484 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939527 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939643 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939664 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "kube-api-access-8wbgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939752 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939809 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939926 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.939958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.939935 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.940763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.940710 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "kube-api-access-5fp6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:02.941594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.941572 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:54:02.942234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.942214 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy" pod "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" (UID: "bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.942328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.942230 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy" pod "c9de4197-e8e0-4abd-a991-d3d9e89e4b45" (UID: "c9de4197-e8e0-4abd-a991-d3d9e89e4b45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:02.964862 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:02.964837 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:54:03.039595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039559 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.039595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039598 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.039928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039672 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.039928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039701 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.039928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039797 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.039928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039910 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.040166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039978 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:03.040166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.039997 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.040166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.040084 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.040343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.040275 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket\") pod \"8f266285-791c-467e-aba7-801f8203f39a\" (UID: \"8f266285-791c-467e-aba7-801f8203f39a\") " Nov 02 17:54:03.040470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.040429 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:03.040748 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.040705 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:03.040927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.040904 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:03.041140 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041117 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041146 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041160 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041174 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041189 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041203 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041217 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041256 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041271 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fp6p\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-kube-api-access-5fp6p\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041285 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041297 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041311 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041323 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041335 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041350 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041363 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbgw\" (UniqueName: \"kubernetes.io/projected/c9de4197-e8e0-4abd-a991-d3d9e89e4b45-kube-api-access-8wbgw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.041481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041377 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f266285-791c-467e-aba7-801f8203f39a-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.042155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.041590 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:03.042620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.042592 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "kube-api-access-cv598". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:03.042870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.042846 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:54:03.043232 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.043213 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:03.044018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.043998 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy" pod "8f266285-791c-467e-aba7-801f8203f39a" (UID: "8f266285-791c-467e-aba7-801f8203f39a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:03.143451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.143416 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f266285-791c-467e-aba7-801f8203f39a-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.143451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.143445 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.143451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.143459 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.143809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.143472 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv598\" (UniqueName: \"kubernetes.io/projected/8f266285-791c-467e-aba7-801f8203f39a-kube-api-access-cv598\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.143809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.143485 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f266285-791c-467e-aba7-801f8203f39a-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:03.335463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.335427 5647 generic.go:356] "Generic (PLEG): container finished" podID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerID="7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3" exitCode=0 Nov 02 17:54:03.335673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.335495 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" Nov 02 17:54:03.335673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.335513 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" event={"ID":"c9de4197-e8e0-4abd-a991-d3d9e89e4b45","Type":"ContainerDied","Data":"7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3"} Nov 02 17:54:03.335673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.335572 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw" event={"ID":"c9de4197-e8e0-4abd-a991-d3d9e89e4b45","Type":"ContainerDied","Data":"18b7189ecd19c360981c0f8e001f27f18603d9510369df39132768c232c3da98"} Nov 02 17:54:03.335673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.335594 5647 scope.go:122] "RemoveContainer" containerID="7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3" Nov 02 17:54:03.337349 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.337322 5647 generic.go:356] "Generic (PLEG): container finished" podID="8f266285-791c-467e-aba7-801f8203f39a" containerID="f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3" exitCode=0 Nov 02 17:54:03.337469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.337455 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" Nov 02 17:54:03.337589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.337562 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" event={"ID":"8f266285-791c-467e-aba7-801f8203f39a","Type":"ContainerDied","Data":"f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3"} Nov 02 17:54:03.337708 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.337620 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk" event={"ID":"8f266285-791c-467e-aba7-801f8203f39a","Type":"ContainerDied","Data":"f055eefbb43d7b99573eab8e23bfc9d5c7e787250d634cf1073650c96407c96c"} Nov 02 17:54:03.339146 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.339111 5647 generic.go:356] "Generic (PLEG): container finished" podID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerID="934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300" exitCode=0 Nov 02 17:54:03.339234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.339165 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" Nov 02 17:54:03.339234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.339195 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" event={"ID":"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9","Type":"ContainerDied","Data":"934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300"} Nov 02 17:54:03.339363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.339237 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs" event={"ID":"bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9","Type":"ContainerDied","Data":"f5ce669512ce7bcbad7f64fb65dbda72034c0daf485d61552d81e10db280b132"} Nov 02 17:54:03.347827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.347807 5647 scope.go:122] "RemoveContainer" containerID="7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3" Nov 02 17:54:03.348151 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:03.348131 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3\": container with ID starting with 7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3 not found: ID does not exist" containerID="7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3" Nov 02 17:54:03.348204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.348161 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3"} err="failed to get container status \"7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3\": rpc error: code = NotFound desc = could not find container \"7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3\": container with ID starting with 7589fd9a654fec4d4d0f82bbce81f9ad4d589bbb1aa25c12dfec7304bc922eb3 not found: ID does not exist" Nov 02 17:54:03.348204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.348183 5647 scope.go:122] "RemoveContainer" containerID="f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3" Nov 02 17:54:03.368928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.368903 5647 scope.go:122] "RemoveContainer" containerID="f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3" Nov 02 17:54:03.369281 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:03.369262 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3\": container with ID starting with f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3 not found: ID does not exist" containerID="f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3" Nov 02 17:54:03.369324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.369291 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3"} err="failed to get container status \"f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3\": rpc error: code = NotFound desc = could not find container \"f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3\": container with ID starting with f45b8f96efe098699f6539fcdc3c7e4f306869032b0ffc41d1ef132b799bd1c3 not found: ID does not exist" Nov 02 17:54:03.369324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.369310 5647 scope.go:122] "RemoveContainer" containerID="934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300" Nov 02 17:54:03.379505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.379485 5647 scope.go:122] "RemoveContainer" containerID="934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300" Nov 02 17:54:03.379812 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:03.379790 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300\": container with ID starting with 934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300 not found: ID does not exist" containerID="934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300" Nov 02 17:54:03.379878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.379826 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300"} err="failed to get container status \"934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300\": rpc error: code = NotFound desc = could not find container \"934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300\": container with ID starting with 934846d772f3ea439b9d3bee8c818a3ee94444a78098201ece79cda031c72300 not found: ID does not exist" Nov 02 17:54:03.504019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.503979 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs"] Nov 02 17:54:03.509205 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.509170 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-btngs"] Nov 02 17:54:03.619878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.619797 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw"] Nov 02 17:54:03.624601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.624573 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqqgw"] Nov 02 17:54:03.736310 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.736275 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk"] Nov 02 17:54:03.741291 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:03.741259 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-6t4wk"] Nov 02 17:54:05.081388 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:05.081349 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f266285-791c-467e-aba7-801f8203f39a" path="/var/lib/kubelet/pods/8f266285-791c-467e-aba7-801f8203f39a/volumes" Nov 02 17:54:05.081849 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:05.081840 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" path="/var/lib/kubelet/pods/bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9/volumes" Nov 02 17:54:05.082255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:05.082240 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" path="/var/lib/kubelet/pods/c9de4197-e8e0-4abd-a991-d3d9e89e4b45/volumes" Nov 02 17:54:05.965678 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:05.965633 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:54:09.668426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:09.668384 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt"] Nov 02 17:54:09.668947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:09.668691 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" containerID="cri-o://a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae" gracePeriod=30 Nov 02 17:54:12.542386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:12.542341 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:54:14.667281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:14.667247 5647 patch_prober.go:29] interesting pod/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:54:14.667785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:14.667750 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:54:15.002764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.002030 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:54:15.172633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172602 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172699 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172755 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172828 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172799 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172833 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172875 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172911 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.172999 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.172990 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.173237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173042 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs\") pod \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\" (UID: \"5171dfce-2108-4f03-8b01-c0fd91b3e5a3\") " Nov 02 17:54:15.173237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173056 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:15.173237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173063 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:15.173453 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173429 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:15.173706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173548 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:15.173706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173604 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:15.173912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173843 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.173912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173856 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.173912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173866 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.173912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173878 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.173912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.173891 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.175822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.175783 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:15.175822 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.175794 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:54:15.175986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.175911 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:54:15.175986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.175960 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j" pod "5171dfce-2108-4f03-8b01-c0fd91b3e5a3" (UID: "5171dfce-2108-4f03-8b01-c0fd91b3e5a3"). InnerVolumeSpecName "kube-api-access-lbf7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:15.275681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.275596 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.275681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.275628 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbf7j\" (UniqueName: \"kubernetes.io/projected/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-kube-api-access-lbf7j\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.275681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.275640 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.275681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.275649 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5171dfce-2108-4f03-8b01-c0fd91b3e5a3-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:15.447710 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.447671 5647 generic.go:356] "Generic (PLEG): container finished" podID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerID="a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae" exitCode=0 Nov 02 17:54:15.447916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.447768 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" Nov 02 17:54:15.447916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.447760 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" event={"ID":"5171dfce-2108-4f03-8b01-c0fd91b3e5a3","Type":"ContainerDied","Data":"a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae"} Nov 02 17:54:15.447916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.447892 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt" event={"ID":"5171dfce-2108-4f03-8b01-c0fd91b3e5a3","Type":"ContainerDied","Data":"78506386c589aa48eea0358f62eccc6826a904bed359e0f4668f4734b859a2e6"} Nov 02 17:54:15.448033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.447923 5647 scope.go:122] "RemoveContainer" containerID="a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae" Nov 02 17:54:15.459230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.459202 5647 scope.go:122] "RemoveContainer" containerID="b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca" Nov 02 17:54:15.468870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.468849 5647 scope.go:122] "RemoveContainer" containerID="a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae" Nov 02 17:54:15.469120 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:15.469102 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae\": container with ID starting with a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae not found: ID does not exist" containerID="a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae" Nov 02 17:54:15.469169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.469130 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae"} err="failed to get container status \"a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae\": rpc error: code = NotFound desc = could not find container \"a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae\": container with ID starting with a8f73b126dbd4c6fd9cf7897acfb042a60ddc77da3d187a78ac4f8b7e3d7baae not found: ID does not exist" Nov 02 17:54:15.469169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.469147 5647 scope.go:122] "RemoveContainer" containerID="b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca" Nov 02 17:54:15.469359 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:15.469343 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca\": container with ID starting with b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca not found: ID does not exist" containerID="b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca" Nov 02 17:54:15.469401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.469364 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca"} err="failed to get container status \"b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca\": rpc error: code = NotFound desc = could not find container \"b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca\": container with ID starting with b6406b484b29660798cf2f835ff5cd8fbe213aeda192f247d7a59cca2143a1ca not found: ID does not exist" Nov 02 17:54:15.645164 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.645131 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt"] Nov 02 17:54:15.651437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:15.651412 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-27ab38-openshift-default-64975f7f6f-t42mt"] Nov 02 17:54:17.088515 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:17.088475 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" path="/var/lib/kubelet/pods/5171dfce-2108-4f03-8b01-c0fd91b3e5a3/volumes" Nov 02 17:54:22.509448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:22.509415 5647 generic.go:356] "Generic (PLEG): container finished" podID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerID="aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9" exitCode=0 Nov 02 17:54:22.509851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:22.509489 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" event={"ID":"10786ae4-769f-4e4e-a01a-e0354e2366bd","Type":"ContainerDied","Data":"aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9"} Nov 02 17:54:22.509951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:22.509935 5647 scope.go:122] "RemoveContainer" containerID="aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9" Nov 02 17:54:22.511577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:22.511542 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:54:22.511657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:22.511639 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:54:22.511694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:22.511678 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:54:23.523758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:23.523406 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" event={"ID":"10786ae4-769f-4e4e-a01a-e0354e2366bd","Type":"ContainerStarted","Data":"7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a"} Nov 02 17:54:23.954507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:23.954464 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:54:23.954696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:23.954606 5647 patch_prober.go:29] interesting pod/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:54:23.954696 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:23.954658 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 17:54:24.955070 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:24.955028 5647 patch_prober.go:29] interesting pod/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:54:24.955611 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:24.955096 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 17:54:25.955151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:25.955115 5647 patch_prober.go:29] interesting pod/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:54:25.955654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:25.955180 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 17:54:26.955171 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:26.955133 5647 patch_prober.go:29] interesting pod/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:54:26.955607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:26.955201 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 17:54:27.954984 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:27.954947 5647 patch_prober.go:29] interesting pod/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Nov 02 17:54:27.955188 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:27.955014 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Nov 02 17:54:28.959773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:28.959729 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:54:28.960191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:28.960136 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:54:28.960981 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:28.960961 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:54:35.804843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:35.804812 5647 scope.go:122] "RemoveContainer" containerID="0f05b3bd0c1260d6f0904001984846536a1c4899403ce950a6bd3a75247b1c9a" Nov 02 17:54:35.814951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:35.814922 5647 scope.go:122] "RemoveContainer" containerID="7eff6cce986137c99379ebe87818100a256af1b8f92083420477ac33ab61efcf" Nov 02 17:54:35.824424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:35.824398 5647 scope.go:122] "RemoveContainer" containerID="5660a09db49b9de8930c2fb7263b3cc03631c05b86fef930c811d09ef8e340f5" Nov 02 17:54:35.947589 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:35.947551 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:54:42.117285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:42.117235 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6"] Nov 02 17:54:47.117964 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.117919 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" podUID="ec1b15a3-a48f-4722-9976-66b8a0472947" containerName="payload-pre-processing" containerID="cri-o://6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b" gracePeriod=25 Nov 02 17:54:47.415618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.415545 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:54:47.468247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.468212 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume\") pod \"ec1b15a3-a48f-4722-9976-66b8a0472947\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " Nov 02 17:54:47.468419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.468280 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb\" (UniqueName: \"kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb\") pod \"ec1b15a3-a48f-4722-9976-66b8a0472947\" (UID: \"ec1b15a3-a48f-4722-9976-66b8a0472947\") " Nov 02 17:54:47.468635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.468597 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume" pod "ec1b15a3-a48f-4722-9976-66b8a0472947" (UID: "ec1b15a3-a48f-4722-9976-66b8a0472947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:54:47.468794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.468777 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1b15a3-a48f-4722-9976-66b8a0472947-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.470589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.470563 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb" pod "ec1b15a3-a48f-4722-9976-66b8a0472947" (UID: "ec1b15a3-a48f-4722-9976-66b8a0472947"). InnerVolumeSpecName "kube-api-access-vmpvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:54:47.569886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.569851 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmpvb\" (UniqueName: \"kubernetes.io/projected/ec1b15a3-a48f-4722-9976-66b8a0472947-kube-api-access-vmpvb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:54:47.701373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.701285 5647 generic.go:356] "Generic (PLEG): container finished" podID="ec1b15a3-a48f-4722-9976-66b8a0472947" containerID="6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b" exitCode=0 Nov 02 17:54:47.701373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.701334 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" Nov 02 17:54:47.701594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.701370 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" event={"ID":"ec1b15a3-a48f-4722-9976-66b8a0472947","Type":"ContainerDied","Data":"6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b"} Nov 02 17:54:47.701594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.701428 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6" event={"ID":"ec1b15a3-a48f-4722-9976-66b8a0472947","Type":"ContainerDied","Data":"b2e301da3dc8b754cd94744d42936b7c2d5c47aa5ce7c031b17ca19f2f74aff5"} Nov 02 17:54:47.701594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.701462 5647 scope.go:122] "RemoveContainer" containerID="6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b" Nov 02 17:54:47.712099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.712081 5647 scope.go:122] "RemoveContainer" containerID="6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b" Nov 02 17:54:47.712398 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:54:47.712377 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b\": container with ID starting with 6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b not found: ID does not exist" containerID="6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b" Nov 02 17:54:47.712458 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.712407 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b"} err="failed to get container status \"6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b\": rpc error: code = NotFound desc = could not find container \"6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b\": container with ID starting with 6aa67afd77c8644a076844ec915793ddbf9379ac67e8866badb238bc3a2d562b not found: ID does not exist" Nov 02 17:54:47.869283 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.869243 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6"] Nov 02 17:54:47.874055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:47.874026 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-666de0-558899697f-zfxd6"] Nov 02 17:54:49.082940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:49.082897 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1b15a3-a48f-4722-9976-66b8a0472947" path="/var/lib/kubelet/pods/ec1b15a3-a48f-4722-9976-66b8a0472947/volumes" Nov 02 17:54:55.548103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548063 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89"] Nov 02 17:54:55.548621 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548608 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548623 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548646 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548651 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548658 5647 cpu_manager.go:414] "containerMap: removing container" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548663 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548671 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cb758a75-cf10-4257-a29e-c3f05157d38d" containerName="maas-api" Nov 02 17:54:55.548676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548675 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb758a75-cf10-4257-a29e-c3f05157d38d" containerName="maas-api" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548682 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ec1b15a3-a48f-4722-9976-66b8a0472947" containerName="payload-pre-processing" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548686 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1b15a3-a48f-4722-9976-66b8a0472947" containerName="payload-pre-processing" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548710 5647 cpu_manager.go:414] "containerMap: removing container" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548714 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548723 5647 cpu_manager.go:414] "containerMap: removing container" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548727 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548830 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548837 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="5171dfce-2108-4f03-8b01-c0fd91b3e5a3" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548846 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9de4197-e8e0-4abd-a991-d3d9e89e4b45" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548855 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9aa48f-d3bf-434a-aca7-e1f52c0b68e9" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548861 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f266285-791c-467e-aba7-801f8203f39a" containerName="istio-proxy" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548867 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb758a75-cf10-4257-a29e-c3f05157d38d" containerName="maas-api" Nov 02 17:54:55.548919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.548873 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1b15a3-a48f-4722-9976-66b8a0472947" containerName="payload-pre-processing" Nov 02 17:54:55.552359 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.552342 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.606686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.606654 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-700abe\"" Nov 02 17:54:55.657618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.657583 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-700abe-dockercfg-qlg5h\"" Nov 02 17:54:55.661426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.661400 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89"] Nov 02 17:54:55.687216 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.687182 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xss\" (UniqueName: \"kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss\") pod \"payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.687375 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.687235 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.789800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.789762 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xss\" (UniqueName: \"kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss\") pod \"payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.790613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.790567 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.792069 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.791959 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.848353 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.848321 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xss\" (UniqueName: \"kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss\") pod \"payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.866351 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.866320 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:55.976790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.976688 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6"] Nov 02 17:54:55.985324 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:55.985296 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.074807 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.043122 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1cea7c-openshift-default-dockercfg-drvt7\"" Nov 02 17:54:56.127985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.121510 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.128297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.128267 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.128469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.128452 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgk6\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.128601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.128587 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.128718 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.128707 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.129005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.128989 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.129126 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.129113 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.129285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.129273 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.129396 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.129385 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.164785 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.138823 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6"] Nov 02 17:54:56.248839 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.248787 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.249208 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.249187 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.250660 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.250636 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgk6\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.250901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.250884 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.251078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.251054 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.251257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.251242 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.251390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.251376 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.251570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.251556 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.251719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.251705 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.252160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.252134 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.252540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.252501 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.252870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.252845 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.253109 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.253090 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.253466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.251253 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.262834 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.260123 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.263185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.263162 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.263603 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.263468 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89"] Nov 02 17:54:56.268629 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:54:56.268597 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f35f43b_a6e2_4870_b563_bbbd8202df8d.slice/crio-b5ff6bb1781ed1259b64a4a52f61419cd349b37ae11fb5b7d13d4a449ee6fa33 WatchSource:0}: Error finding container b5ff6bb1781ed1259b64a4a52f61419cd349b37ae11fb5b7d13d4a449ee6fa33: Status 404 returned error can't find the container with id b5ff6bb1781ed1259b64a4a52f61419cd349b37ae11fb5b7d13d4a449ee6fa33 Nov 02 17:54:56.318097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.318041 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgk6\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.323190 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.320111 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token\") pod \"e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.356331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.356305 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t"] Nov 02 17:54:56.361798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.361777 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.394831 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.394804 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:56.417947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.417916 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t"] Nov 02 17:54:56.455826 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.455681 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzvc\" (UniqueName: \"kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc\") pod \"payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.456242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.456015 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.557448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.557409 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.557448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.557447 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzvc\" (UniqueName: \"kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc\") pod \"payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.558292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.558267 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.615954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.615918 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzvc\" (UniqueName: \"kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc\") pod \"payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.688350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.688259 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:56.785560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.785507 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6"] Nov 02 17:54:56.786112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.786007 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" event={"ID":"1f35f43b-a6e2-4870-b563-bbbd8202df8d","Type":"ContainerStarted","Data":"04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876"} Nov 02 17:54:56.786112 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.786066 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" event={"ID":"1f35f43b-a6e2-4870-b563-bbbd8202df8d","Type":"ContainerStarted","Data":"b5ff6bb1781ed1259b64a4a52f61419cd349b37ae11fb5b7d13d4a449ee6fa33"} Nov 02 17:54:56.786665 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.786329 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:54:56.786987 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:54:56.786959 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7b82c1_e862_4aa5_9032_8c3a6ceb3c20.slice/crio-1d9ffa48d6f6f24905c961615c0db4f3d8a56da4feea7d2de68ce982ccdd4e12 WatchSource:0}: Error finding container 1d9ffa48d6f6f24905c961615c0db4f3d8a56da4feea7d2de68ce982ccdd4e12: Status 404 returned error can't find the container with id 1d9ffa48d6f6f24905c961615c0db4f3d8a56da4feea7d2de68ce982ccdd4e12 Nov 02 17:54:56.789526 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.789487 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:54:56.789630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.789562 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:54:56.789630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:56.789591 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:54:57.072036 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.071888 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" podStartSLOduration=2.071868715 podStartE2EDuration="2.071868715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:56.91331615 +0000 UTC m=+1883.986843352" watchObservedRunningTime="2026-11-02 17:54:57.071868715 +0000 UTC m=+1884.145395919" Nov 02 17:54:57.072966 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.072935 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t"] Nov 02 17:54:57.100934 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:54:57.100884 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9491ac0a_2ae5_48cc_85a1_5856520fb4d5.slice/crio-6304799334bdfdd8c5378535b8596c454d157d9d4898df30ed935f5d8cfd9fe0 WatchSource:0}: Error finding container 6304799334bdfdd8c5378535b8596c454d157d9d4898df30ed935f5d8cfd9fe0: Status 404 returned error can't find the container with id 6304799334bdfdd8c5378535b8596c454d157d9d4898df30ed935f5d8cfd9fe0 Nov 02 17:54:57.794578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.794540 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" event={"ID":"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20","Type":"ContainerStarted","Data":"612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1"} Nov 02 17:54:57.795102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.794607 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" event={"ID":"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20","Type":"ContainerStarted","Data":"1d9ffa48d6f6f24905c961615c0db4f3d8a56da4feea7d2de68ce982ccdd4e12"} Nov 02 17:54:57.796330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.796297 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" event={"ID":"9491ac0a-2ae5-48cc-85a1-5856520fb4d5","Type":"ContainerStarted","Data":"ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07"} Nov 02 17:54:57.796466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.796365 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:54:57.796514 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.796465 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" event={"ID":"9491ac0a-2ae5-48cc-85a1-5856520fb4d5","Type":"ContainerStarted","Data":"6304799334bdfdd8c5378535b8596c454d157d9d4898df30ed935f5d8cfd9fe0"} Nov 02 17:54:57.920097 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:57.920046 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podStartSLOduration=2.920032043 podStartE2EDuration="2.920032043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:57.917109448 +0000 UTC m=+1884.990636663" watchObservedRunningTime="2026-11-02 17:54:57.920032043 +0000 UTC m=+1884.993559245" Nov 02 17:54:58.395151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:58.395120 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:58.401584 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:58.401552 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:58.523993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:58.523938 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" podStartSLOduration=2.5239236480000002 podStartE2EDuration="2.523923648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:58.056485349 +0000 UTC m=+1885.130012554" watchObservedRunningTime="2026-11-02 17:54:58.523923648 +0000 UTC m=+1885.597450851" Nov 02 17:54:58.803075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:58.802973 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:54:58.803840 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:54:58.803823 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:55:05.948316 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:05.948271 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:55:07.799451 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:07.799420 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:55:08.805567 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:08.805537 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:55:11.945043 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:11.944966 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn"] Nov 02 17:55:11.949631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:11.949608 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.106805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.073866 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-9ac699-openshift-default-dockercfg-lngtg\"" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118186 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118237 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118400 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118451 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118534 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118571 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118601 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5x49\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118633 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.120538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.118702 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.127212 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.127181 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn"] Nov 02 17:55:12.220671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220592 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.220671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220643 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.220962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220698 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.220962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220724 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.220962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220788 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5x49\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.220962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220817 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.220962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.220872 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.221285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.221064 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.221285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.221089 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.221285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.221271 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.221459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.221438 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.221777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.221689 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.221870 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.221824 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.222047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.222023 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.223847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.223822 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.223988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.223830 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.279524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.279493 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5x49\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.279524 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.279512 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token\") pod \"e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.286609 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.286580 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:12.779777 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.779709 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn"] Nov 02 17:55:12.780627 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:55:12.780597 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3cf331_f2be_42c5_8866_f9cf477086f5.slice/crio-46891db4022459fa03f26766a90e24583b0f3e73551bf21e061d10fcae783255 WatchSource:0}: Error finding container 46891db4022459fa03f26766a90e24583b0f3e73551bf21e061d10fcae783255: Status 404 returned error can't find the container with id 46891db4022459fa03f26766a90e24583b0f3e73551bf21e061d10fcae783255 Nov 02 17:55:12.783328 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.783295 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:55:12.783405 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.783365 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:55:12.783405 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.783393 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:55:12.963886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.963852 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" event={"ID":"ff3cf331-f2be-42c5-8866-f9cf477086f5","Type":"ContainerStarted","Data":"2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e"} Nov 02 17:55:12.964377 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:12.963907 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" event={"ID":"ff3cf331-f2be-42c5-8866-f9cf477086f5","Type":"ContainerStarted","Data":"46891db4022459fa03f26766a90e24583b0f3e73551bf21e061d10fcae783255"} Nov 02 17:55:13.084725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:13.084606 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podStartSLOduration=2.084589471 podStartE2EDuration="2.084589471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:13.081903401 +0000 UTC m=+1900.155430609" watchObservedRunningTime="2026-11-02 17:55:13.084589471 +0000 UTC m=+1900.158116673" Nov 02 17:55:13.286805 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:13.286767 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:13.292448 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:13.292418 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:13.970780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:13.970744 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:13.971727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:13.971708 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:55:24.366719 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.366685 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk"] Nov 02 17:55:24.370620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.370601 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.430194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.430163 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ccf013-serving-cert\"" Nov 02 17:55:24.484201 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.484160 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk"] Nov 02 17:55:24.557988 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.557949 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.558166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.558077 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.558255 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.558226 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.558321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.558307 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv9df\" (UniqueName: \"kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.666816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.665874 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv9df\" (UniqueName: \"kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.666816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.665994 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.666816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.666072 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.666816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.666222 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.669252 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.669011 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.674582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.674554 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.679643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.679610 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.725466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.725432 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv9df\" (UniqueName: \"kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df\") pod \"maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:24.985646 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:24.985556 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:25.366487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:25.366455 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk"] Nov 02 17:55:25.368090 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:55:25.368057 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9c2867_5a09_4566_9dec_d0872e123f12.slice/crio-cda2a69f11518542babe66be368d0bc40e48739060ac6af794273fa214dcff8a WatchSource:0}: Error finding container cda2a69f11518542babe66be368d0bc40e48739060ac6af794273fa214dcff8a: Status 404 returned error can't find the container with id cda2a69f11518542babe66be368d0bc40e48739060ac6af794273fa214dcff8a Nov 02 17:55:26.057357 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:26.057320 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" event={"ID":"9b9c2867-5a09-4566-9dec-d0872e123f12","Type":"ContainerStarted","Data":"cda2a69f11518542babe66be368d0bc40e48739060ac6af794273fa214dcff8a"} Nov 02 17:55:27.067067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:27.067030 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" event={"ID":"9b9c2867-5a09-4566-9dec-d0872e123f12","Type":"ContainerStarted","Data":"61da15bf67adcd77b47f0a1b7e27c3b0e50041fe59c120d88cb60b3c2d5c83f2"} Nov 02 17:55:27.067557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:27.067111 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:27.209996 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:27.209946 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" podStartSLOduration=2.252532398 podStartE2EDuration="3.209931268s" totalImagesPullingTime="957.39887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:27.207101373 +0000 UTC m=+1914.280628576" watchObservedRunningTime="2026-11-02 17:55:27.209931268 +0000 UTC m=+1914.283458519" Nov 02 17:55:28.837671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:28.837625 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx"] Nov 02 17:55:28.838226 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:28.837902 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" podUID="e7ea30ba-2a57-4111-ab0e-c8b9eb458785" containerName="maas-api" containerID="cri-o://dd271de8bb6f7fe9a704954aa033e890f86b3b1217bab946605a412715e45080" gracePeriod=30 Nov 02 17:55:29.146980 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.146904 5647 generic.go:356] "Generic (PLEG): container finished" podID="e7ea30ba-2a57-4111-ab0e-c8b9eb458785" containerID="dd271de8bb6f7fe9a704954aa033e890f86b3b1217bab946605a412715e45080" exitCode=0 Nov 02 17:55:29.149446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.149414 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" event={"ID":"e7ea30ba-2a57-4111-ab0e-c8b9eb458785","Type":"ContainerDied","Data":"dd271de8bb6f7fe9a704954aa033e890f86b3b1217bab946605a412715e45080"} Nov 02 17:55:29.174813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.174789 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:55:29.238193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.238160 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca\") pod \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " Nov 02 17:55:29.238402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.238203 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c\" (UniqueName: \"kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c\") pod \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " Nov 02 17:55:29.238402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.238241 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs\") pod \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " Nov 02 17:55:29.238402 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.238374 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls\") pod \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\" (UID: \"e7ea30ba-2a57-4111-ab0e-c8b9eb458785\") " Nov 02 17:55:29.238602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.238572 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca" pod "e7ea30ba-2a57-4111-ab0e-c8b9eb458785" (UID: "e7ea30ba-2a57-4111-ab0e-c8b9eb458785"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:29.238973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.238948 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.240724 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.240692 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls" pod "e7ea30ba-2a57-4111-ab0e-c8b9eb458785" (UID: "e7ea30ba-2a57-4111-ab0e-c8b9eb458785"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:29.241064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.241042 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs" pod "e7ea30ba-2a57-4111-ab0e-c8b9eb458785" (UID: "e7ea30ba-2a57-4111-ab0e-c8b9eb458785"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:29.241064 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.241053 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c" pod "e7ea30ba-2a57-4111-ab0e-c8b9eb458785" (UID: "e7ea30ba-2a57-4111-ab0e-c8b9eb458785"). InnerVolumeSpecName "kube-api-access-n776c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:29.339801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.339762 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.339801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.339803 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.340026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.339816 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n776c\" (UniqueName: \"kubernetes.io/projected/e7ea30ba-2a57-4111-ab0e-c8b9eb458785-kube-api-access-n776c\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.348531 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.348499 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv"] Nov 02 17:55:29.348727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.348690 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" podUID="f189dc21-699c-4947-84c6-200e7f4adc4a" containerName="payload-processing" containerID="cri-o://54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392" gracePeriod=30 Nov 02 17:55:29.402965 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.402896 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k"] Nov 02 17:55:29.403099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.403076 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" podUID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" containerName="payload-pre-processing" containerID="cri-o://47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e" gracePeriod=30 Nov 02 17:55:29.675421 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.675383 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:55:29.706694 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.706668 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:55:29.745484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.745450 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume\") pod \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " Nov 02 17:55:29.745663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.745587 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq\" (UniqueName: \"kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq\") pod \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\" (UID: \"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27\") " Nov 02 17:55:29.745663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.745639 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume\") pod \"f189dc21-699c-4947-84c6-200e7f4adc4a\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " Nov 02 17:55:29.745790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.745710 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h\" (UniqueName: \"kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h\") pod \"f189dc21-699c-4947-84c6-200e7f4adc4a\" (UID: \"f189dc21-699c-4947-84c6-200e7f4adc4a\") " Nov 02 17:55:29.745859 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.745830 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume" pod "4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" (UID: "4523c9b1-2b78-4b6f-9f06-e93a92f8fc27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:29.746108 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.746084 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume" pod "f189dc21-699c-4947-84c6-200e7f4adc4a" (UID: "f189dc21-699c-4947-84c6-200e7f4adc4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:29.746314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.746293 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f189dc21-699c-4947-84c6-200e7f4adc4a-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.746361 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.746318 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.747918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.747898 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq" pod "4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" (UID: "4523c9b1-2b78-4b6f-9f06-e93a92f8fc27"). InnerVolumeSpecName "kube-api-access-7grfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:29.747918 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.747908 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h" pod "f189dc21-699c-4947-84c6-200e7f4adc4a" (UID: "f189dc21-699c-4947-84c6-200e7f4adc4a"). InnerVolumeSpecName "kube-api-access-7wn6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:29.865758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.847966 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7grfq\" (UniqueName: \"kubernetes.io/projected/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27-kube-api-access-7grfq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:29.865758 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:29.848018 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wn6h\" (UniqueName: \"kubernetes.io/projected/f189dc21-699c-4947-84c6-200e7f4adc4a-kube-api-access-7wn6h\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:30.156663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.156630 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" Nov 02 17:55:30.156908 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.156669 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx" event={"ID":"e7ea30ba-2a57-4111-ab0e-c8b9eb458785","Type":"ContainerDied","Data":"0360e6ec668a0a8b5eaa296c669fd02a06d647cd85c2d46bb1be4596e5978486"} Nov 02 17:55:30.156908 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.156765 5647 scope.go:122] "RemoveContainer" containerID="dd271de8bb6f7fe9a704954aa033e890f86b3b1217bab946605a412715e45080" Nov 02 17:55:30.158317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.158291 5647 generic.go:356] "Generic (PLEG): container finished" podID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" containerID="47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e" exitCode=0 Nov 02 17:55:30.158420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.158366 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" Nov 02 17:55:30.158482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.158445 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" event={"ID":"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27","Type":"ContainerDied","Data":"47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e"} Nov 02 17:55:30.158670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.158644 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" event={"ID":"4523c9b1-2b78-4b6f-9f06-e93a92f8fc27","Type":"ContainerDied","Data":"9917aae436770ee8777e22c3ab04f57f87143f8b8605fb98efe344d7805d873d"} Nov 02 17:55:30.160829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.160801 5647 generic.go:356] "Generic (PLEG): container finished" podID="f189dc21-699c-4947-84c6-200e7f4adc4a" containerID="54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392" exitCode=0 Nov 02 17:55:30.160973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.160847 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" Nov 02 17:55:30.160973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.160911 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" event={"ID":"f189dc21-699c-4947-84c6-200e7f4adc4a","Type":"ContainerDied","Data":"54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392"} Nov 02 17:55:30.160973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.160949 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" event={"ID":"f189dc21-699c-4947-84c6-200e7f4adc4a","Type":"ContainerDied","Data":"846e443c6eab83eba0468e5d3da9dbbca4996102d9af82be3ae9207243231340"} Nov 02 17:55:30.169989 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.169665 5647 scope.go:122] "RemoveContainer" containerID="47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e" Nov 02 17:55:30.179182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.179155 5647 scope.go:122] "RemoveContainer" containerID="47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e" Nov 02 17:55:30.179465 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:30.179444 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e\": container with ID starting with 47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e not found: ID does not exist" containerID="47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e" Nov 02 17:55:30.179541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.179473 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e"} err="failed to get container status \"47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e\": rpc error: code = NotFound desc = could not find container \"47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e\": container with ID starting with 47659590c9da02f0a006d2b696444a5b4129c0f339f9bda6055e040dccc6eb2e not found: ID does not exist" Nov 02 17:55:30.179541 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.179491 5647 scope.go:122] "RemoveContainer" containerID="54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392" Nov 02 17:55:30.188496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.188473 5647 scope.go:122] "RemoveContainer" containerID="54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392" Nov 02 17:55:30.188791 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:30.188765 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392\": container with ID starting with 54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392 not found: ID does not exist" containerID="54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392" Nov 02 17:55:30.188888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.188799 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392"} err="failed to get container status \"54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392\": rpc error: code = NotFound desc = could not find container \"54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392\": container with ID starting with 54b70bdf7a389c403b0ade0d858770414761e3b3867bf75346fa4b02b25e8392 not found: ID does not exist" Nov 02 17:55:30.326239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.326207 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx"] Nov 02 17:55:30.329104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.329079 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6f23bd-85f5b67d76-2wzmx"] Nov 02 17:55:30.439423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.439333 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k"] Nov 02 17:55:30.442431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.442408 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k"] Nov 02 17:55:30.553305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.553260 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv"] Nov 02 17:55:30.557299 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.557266 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv"] Nov 02 17:55:30.659470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.659427 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6f23bd-86954875c9-cwt9k" podUID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.153:9004: i/o timeout" Nov 02 17:55:30.659628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:30.659442 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-6f23bd-7f7785f95b-p96pv" podUID="f189dc21-699c-4947-84c6-200e7f4adc4a" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.154:9004: i/o timeout" Nov 02 17:55:31.082423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:31.082389 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" path="/var/lib/kubelet/pods/4523c9b1-2b78-4b6f-9f06-e93a92f8fc27/volumes" Nov 02 17:55:31.082880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:31.082866 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ea30ba-2a57-4111-ab0e-c8b9eb458785" path="/var/lib/kubelet/pods/e7ea30ba-2a57-4111-ab0e-c8b9eb458785/volumes" Nov 02 17:55:31.083220 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:31.083208 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f189dc21-699c-4947-84c6-200e7f4adc4a" path="/var/lib/kubelet/pods/f189dc21-699c-4947-84c6-200e7f4adc4a/volumes" Nov 02 17:55:32.952102 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:32.952065 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc"] Nov 02 17:55:32.952550 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:32.952343 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" containerID="cri-o://3b44dd15c882bbaa72b9f328106ad8922af3de1a30756a15d43aef6ed3399fac" gracePeriod=30 Nov 02 17:55:32.960145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:32.960108 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7"] Nov 02 17:55:32.960360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:32.960332 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" containerID="cri-o://f19e37e25618c4b65c25d8a3d421cfed1ba10dde4b68ff8e67f41ba954812d5b" gracePeriod=30 Nov 02 17:55:32.983371 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:32.983332 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q"] Nov 02 17:55:32.983577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:32.983556 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" containerID="cri-o://53d39cb52331f7bae20e20f9736129a5701a21a4fb4c5d2c49de290c445de38e" gracePeriod=30 Nov 02 17:55:33.174487 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:33.169578 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 17:55:35.947773 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:35.947702 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:55:38.229264 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.229174 5647 generic.go:356] "Generic (PLEG): container finished" podID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerID="f19e37e25618c4b65c25d8a3d421cfed1ba10dde4b68ff8e67f41ba954812d5b" exitCode=0 Nov 02 17:55:38.229699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.229273 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" event={"ID":"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8","Type":"ContainerDied","Data":"f19e37e25618c4b65c25d8a3d421cfed1ba10dde4b68ff8e67f41ba954812d5b"} Nov 02 17:55:38.231380 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.231358 5647 generic.go:356] "Generic (PLEG): container finished" podID="76250067-6ed8-4158-8363-30bac127a046" containerID="3b44dd15c882bbaa72b9f328106ad8922af3de1a30756a15d43aef6ed3399fac" exitCode=0 Nov 02 17:55:38.231522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.231438 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" event={"ID":"76250067-6ed8-4158-8363-30bac127a046","Type":"ContainerDied","Data":"3b44dd15c882bbaa72b9f328106ad8922af3de1a30756a15d43aef6ed3399fac"} Nov 02 17:55:38.233875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.233821 5647 generic.go:356] "Generic (PLEG): container finished" podID="a0753785-7233-4533-a398-0059cd0b8758" containerID="53d39cb52331f7bae20e20f9736129a5701a21a4fb4c5d2c49de290c445de38e" exitCode=0 Nov 02 17:55:38.233998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.233882 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" event={"ID":"a0753785-7233-4533-a398-0059cd0b8758","Type":"ContainerDied","Data":"53d39cb52331f7bae20e20f9736129a5701a21a4fb4c5d2c49de290c445de38e"} Nov 02 17:55:38.233998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.233937 5647 scope.go:122] "RemoveContainer" containerID="9aababaca92a01c536307e4c65145139419eac16cfe4ed272f9e8957311c9d13" Nov 02 17:55:38.274213 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.274187 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:55:38.310714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.310686 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:55:38.313327 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.313303 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:55:38.345819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.345785 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.345819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.345821 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.345855 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.345907 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.346063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.345950 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.345983 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346018 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346098 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346117 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346148 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346160 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346278 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346317 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346325 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346372 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346445 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346478 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346486 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346519 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346597 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346630 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346689 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346728 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346802 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346835 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.346888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346877 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.347709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346910 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy\") pod \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\" (UID: \"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8\") " Nov 02 17:55:38.347709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346956 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.347709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.347034 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.347709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.347086 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs\") pod \"76250067-6ed8-4158-8363-30bac127a046\" (UID: \"76250067-6ed8-4158-8363-30bac127a046\") " Nov 02 17:55:38.347709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.347125 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2\") pod \"a0753785-7233-4533-a398-0059cd0b8758\" (UID: \"a0753785-7233-4533-a398-0059cd0b8758\") " Nov 02 17:55:38.348092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.348069 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.348159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.348098 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.348159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.348114 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.348159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.348132 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.348791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346518 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.348791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346700 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.348791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.346771 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.348855 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.348985 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349115 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349006 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.349180 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349118 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349183 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:38.349242 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349228 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349275 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349356 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349312 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.349591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349497 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.349591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349548 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349555 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.349827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.349603 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:38.350199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.350144 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:38.350199 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.350169 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.350898 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.350876 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:38.351359 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.351333 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:38.351479 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.351427 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:38.351951 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.351931 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2" pod "a0753785-7233-4533-a398-0059cd0b8758" (UID: "a0753785-7233-4533-a398-0059cd0b8758"). InnerVolumeSpecName "kube-api-access-t9nh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.352053 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.351947 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq" pod "76250067-6ed8-4158-8363-30bac127a046" (UID: "76250067-6ed8-4158-8363-30bac127a046"). InnerVolumeSpecName "kube-api-access-wxtkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.352177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.352163 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79" pod "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" (UID: "52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8"). InnerVolumeSpecName "kube-api-access-q7h79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:38.449721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449687 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.449721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449717 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76250067-6ed8-4158-8363-30bac127a046-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.449721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449727 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0753785-7233-4533-a398-0059cd0b8758-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.449721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449759 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.449721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449772 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449781 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0753785-7233-4533-a398-0059cd0b8758-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449790 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449798 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449806 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449815 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7h79\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-kube-api-access-q7h79\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449825 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxtkq\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-kube-api-access-wxtkq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449835 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449843 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449852 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0753785-7233-4533-a398-0059cd0b8758-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449860 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449868 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9nh2\" (UniqueName: \"kubernetes.io/projected/a0753785-7233-4533-a398-0059cd0b8758-kube-api-access-t9nh2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449877 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76250067-6ed8-4158-8363-30bac127a046-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449885 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449895 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449903 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76250067-6ed8-4158-8363-30bac127a046-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449911 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76250067-6ed8-4158-8363-30bac127a046-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449919 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:38.450184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:38.449928 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:39.243182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.243088 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" Nov 02 17:55:39.243182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.243123 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q" event={"ID":"a0753785-7233-4533-a398-0059cd0b8758","Type":"ContainerDied","Data":"205cef2bfc1dc6268d298efd9e79a787ec2a08bf78f07d2c9d246a087dd26b74"} Nov 02 17:55:39.243674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.243192 5647 scope.go:122] "RemoveContainer" containerID="53d39cb52331f7bae20e20f9736129a5701a21a4fb4c5d2c49de290c445de38e" Nov 02 17:55:39.244853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.244833 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" Nov 02 17:55:39.244954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.244885 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7" event={"ID":"52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8","Type":"ContainerDied","Data":"c20a90cfe2c53377abc90004cdb8bf8fad8d89e6c67d97920dba15ef36e13f41"} Nov 02 17:55:39.247559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.246632 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" event={"ID":"76250067-6ed8-4158-8363-30bac127a046","Type":"ContainerDied","Data":"65f7eaf5af05295aa6b5815e9c90a3270ed7b253fb66783c83e96ec621137f89"} Nov 02 17:55:39.247559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.246724 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc" Nov 02 17:55:39.256194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.256171 5647 scope.go:122] "RemoveContainer" containerID="f19e37e25618c4b65c25d8a3d421cfed1ba10dde4b68ff8e67f41ba954812d5b" Nov 02 17:55:39.265577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.265557 5647 scope.go:122] "RemoveContainer" containerID="3b44dd15c882bbaa72b9f328106ad8922af3de1a30756a15d43aef6ed3399fac" Nov 02 17:55:39.422508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.412152 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc"] Nov 02 17:55:39.461629 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.461593 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tdtnc"] Nov 02 17:55:39.529237 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.529155 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7"] Nov 02 17:55:39.531437 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.531413 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-nkpc7"] Nov 02 17:55:39.641513 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.641476 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q"] Nov 02 17:55:39.646256 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:39.646229 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6f23bd-openshift-default-665bbcf8df-tlg9q"] Nov 02 17:55:40.499637 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.499600 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk"] Nov 02 17:55:40.500159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.499866 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" podUID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" containerName="maas-api" containerID="cri-o://5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9" gracePeriod=30 Nov 02 17:55:40.830483 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.830455 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:55:40.888660 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.888621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs\") pod \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " Nov 02 17:55:40.888892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.888719 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls\") pod \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " Nov 02 17:55:40.888892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.888792 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca\") pod \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " Nov 02 17:55:40.888892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.888866 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp\" (UniqueName: \"kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp\") pod \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\" (UID: \"010bf490-d048-49d4-b7c9-ac2dcd50c4f3\") " Nov 02 17:55:40.889137 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.889110 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca" pod "010bf490-d048-49d4-b7c9-ac2dcd50c4f3" (UID: "010bf490-d048-49d4-b7c9-ac2dcd50c4f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:40.889493 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.889466 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:40.891184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.891160 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls" pod "010bf490-d048-49d4-b7c9-ac2dcd50c4f3" (UID: "010bf490-d048-49d4-b7c9-ac2dcd50c4f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:40.891278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.891184 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs" pod "010bf490-d048-49d4-b7c9-ac2dcd50c4f3" (UID: "010bf490-d048-49d4-b7c9-ac2dcd50c4f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:55:40.891278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.891199 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp" pod "010bf490-d048-49d4-b7c9-ac2dcd50c4f3" (UID: "010bf490-d048-49d4-b7c9-ac2dcd50c4f3"). InnerVolumeSpecName "kube-api-access-kk8pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:40.990680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.990642 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:40.990680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.990674 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:40.990680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:40.990685 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8pp\" (UniqueName: \"kubernetes.io/projected/010bf490-d048-49d4-b7c9-ac2dcd50c4f3-kube-api-access-kk8pp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:41.081395 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.081314 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" path="/var/lib/kubelet/pods/52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8/volumes" Nov 02 17:55:41.081970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.081909 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76250067-6ed8-4158-8363-30bac127a046" path="/var/lib/kubelet/pods/76250067-6ed8-4158-8363-30bac127a046/volumes" Nov 02 17:55:41.082684 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.082664 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0753785-7233-4533-a398-0059cd0b8758" path="/var/lib/kubelet/pods/a0753785-7233-4533-a398-0059cd0b8758/volumes" Nov 02 17:55:41.278379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.275718 5647 generic.go:356] "Generic (PLEG): container finished" podID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" containerID="5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9" exitCode=0 Nov 02 17:55:41.278379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.275964 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" event={"ID":"010bf490-d048-49d4-b7c9-ac2dcd50c4f3","Type":"ContainerDied","Data":"5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9"} Nov 02 17:55:41.278379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.276026 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" event={"ID":"010bf490-d048-49d4-b7c9-ac2dcd50c4f3","Type":"ContainerDied","Data":"c4fc7e934b9bcbae2527f6685dab91ab3b46cbdc85478c24dcbc0b3dd3cda357"} Nov 02 17:55:41.278379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.276052 5647 scope.go:122] "RemoveContainer" containerID="5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9" Nov 02 17:55:41.278379 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.276190 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" Nov 02 17:55:41.291162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.291129 5647 scope.go:122] "RemoveContainer" containerID="5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9" Nov 02 17:55:41.291557 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:41.291530 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9\": container with ID starting with 5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9 not found: ID does not exist" containerID="5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9" Nov 02 17:55:41.291645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.291573 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9"} err="failed to get container status \"5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9\": rpc error: code = NotFound desc = could not find container \"5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9\": container with ID starting with 5c6f1a4677fbb9a2e91c246753fc6f9f4c4cbf9f19de425adcede7fd0abd06e9 not found: ID does not exist" Nov 02 17:55:41.447096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.447054 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk"] Nov 02 17:55:41.511760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:41.507912 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk"] Nov 02 17:55:42.224950 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.224918 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs"] Nov 02 17:55:42.225166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.225140 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" podUID="bae87f4e-e00e-4e03-acb5-867a3cae39e2" containerName="payload-processing" containerID="cri-o://9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77" gracePeriod=30 Nov 02 17:55:42.281352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.281312 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4"] Nov 02 17:55:42.281623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.281596 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" podUID="49445ce7-b2d9-4714-b340-2e4426ac72f6" containerName="payload-pre-processing" containerID="cri-o://d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f" gracePeriod=30 Nov 02 17:55:42.727505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.727479 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:55:42.730867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.730844 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:55:42.899796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.889357 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw\" (UniqueName: \"kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw\") pod \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " Nov 02 17:55:42.899796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.889437 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume\") pod \"49445ce7-b2d9-4714-b340-2e4426ac72f6\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " Nov 02 17:55:42.899796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.889543 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume\") pod \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\" (UID: \"bae87f4e-e00e-4e03-acb5-867a3cae39e2\") " Nov 02 17:55:42.899796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.889598 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf\" (UniqueName: \"kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf\") pod \"49445ce7-b2d9-4714-b340-2e4426ac72f6\" (UID: \"49445ce7-b2d9-4714-b340-2e4426ac72f6\") " Nov 02 17:55:42.899796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.893105 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume" pod "49445ce7-b2d9-4714-b340-2e4426ac72f6" (UID: "49445ce7-b2d9-4714-b340-2e4426ac72f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:42.899796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.895209 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume" pod "bae87f4e-e00e-4e03-acb5-867a3cae39e2" (UID: "bae87f4e-e00e-4e03-acb5-867a3cae39e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:42.935761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.935367 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf" pod "49445ce7-b2d9-4714-b340-2e4426ac72f6" (UID: "49445ce7-b2d9-4714-b340-2e4426ac72f6"). InnerVolumeSpecName "kube-api-access-92vbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:42.940811 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.939862 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw" pod "bae87f4e-e00e-4e03-acb5-867a3cae39e2" (UID: "bae87f4e-e00e-4e03-acb5-867a3cae39e2"). InnerVolumeSpecName "kube-api-access-cnddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:42.991623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.991584 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49445ce7-b2d9-4714-b340-2e4426ac72f6-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.991623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.991621 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae87f4e-e00e-4e03-acb5-867a3cae39e2-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.991623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.991632 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92vbf\" (UniqueName: \"kubernetes.io/projected/49445ce7-b2d9-4714-b340-2e4426ac72f6-kube-api-access-92vbf\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:42.991905 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:42.991642 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnddw\" (UniqueName: \"kubernetes.io/projected/bae87f4e-e00e-4e03-acb5-867a3cae39e2-kube-api-access-cnddw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:43.082890 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.082847 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" path="/var/lib/kubelet/pods/010bf490-d048-49d4-b7c9-ac2dcd50c4f3/volumes" Nov 02 17:55:43.294124 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.294094 5647 generic.go:356] "Generic (PLEG): container finished" podID="bae87f4e-e00e-4e03-acb5-867a3cae39e2" containerID="9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77" exitCode=0 Nov 02 17:55:43.294333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.294149 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" Nov 02 17:55:43.294333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.294184 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" event={"ID":"bae87f4e-e00e-4e03-acb5-867a3cae39e2","Type":"ContainerDied","Data":"9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77"} Nov 02 17:55:43.294333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.294258 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs" event={"ID":"bae87f4e-e00e-4e03-acb5-867a3cae39e2","Type":"ContainerDied","Data":"f17f380ccf5ec321bb7c3d6c02f0cc6b6386512ce0fe69f740ee4db533c3f57f"} Nov 02 17:55:43.294333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.294292 5647 scope.go:122] "RemoveContainer" containerID="9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77" Nov 02 17:55:43.296423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.296395 5647 generic.go:356] "Generic (PLEG): container finished" podID="49445ce7-b2d9-4714-b340-2e4426ac72f6" containerID="d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f" exitCode=0 Nov 02 17:55:43.296548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.296448 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" Nov 02 17:55:43.296548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.296461 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" event={"ID":"49445ce7-b2d9-4714-b340-2e4426ac72f6","Type":"ContainerDied","Data":"d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f"} Nov 02 17:55:43.296548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.296505 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4" event={"ID":"49445ce7-b2d9-4714-b340-2e4426ac72f6","Type":"ContainerDied","Data":"6543fbaf040f9c0aae906b90f7c67e1a9ef81992f8510fa24247485502be2063"} Nov 02 17:55:43.305078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.305049 5647 scope.go:122] "RemoveContainer" containerID="9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77" Nov 02 17:55:43.305377 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:43.305359 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77\": container with ID starting with 9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77 not found: ID does not exist" containerID="9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77" Nov 02 17:55:43.305433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.305386 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77"} err="failed to get container status \"9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77\": rpc error: code = NotFound desc = could not find container \"9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77\": container with ID starting with 9ff6513a2598bfa73829b7bc2eee1422149fd61ea4232a3212647c433c042d77 not found: ID does not exist" Nov 02 17:55:43.305433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.305404 5647 scope.go:122] "RemoveContainer" containerID="d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f" Nov 02 17:55:43.314991 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.314970 5647 scope.go:122] "RemoveContainer" containerID="d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f" Nov 02 17:55:43.315295 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:43.315276 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f\": container with ID starting with d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f not found: ID does not exist" containerID="d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f" Nov 02 17:55:43.315335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.315304 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f"} err="failed to get container status \"d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f\": rpc error: code = NotFound desc = could not find container \"d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f\": container with ID starting with d2d6f819cd63aa952455408a0b0ea1bb7d81c9e9a594cf16f02acf21f1a53c7f not found: ID does not exist" Nov 02 17:55:43.460916 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.460884 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs"] Nov 02 17:55:43.464545 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.464517 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fa615-674bb8f4c4-znrzs"] Nov 02 17:55:43.575358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.575269 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4"] Nov 02 17:55:43.580233 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.580206 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fa615-846cfdc9-swxs4"] Nov 02 17:55:43.771666 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:43.771613 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-666de0-76fdb96cc7-g76mk" podUID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.155:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 17:55:45.081668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:45.081632 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49445ce7-b2d9-4714-b340-2e4426ac72f6" path="/var/lib/kubelet/pods/49445ce7-b2d9-4714-b340-2e4426ac72f6/volumes" Nov 02 17:55:45.082091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:45.082075 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae87f4e-e00e-4e03-acb5-867a3cae39e2" path="/var/lib/kubelet/pods/bae87f4e-e00e-4e03-acb5-867a3cae39e2/volumes" Nov 02 17:55:47.547188 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:47.547150 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q"] Nov 02 17:55:47.547650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:47.547410 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" containerID="cri-o://56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f" gracePeriod=30 Nov 02 17:55:49.671793 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.671751 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk"] Nov 02 17:55:49.672555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672534 5647 cpu_manager.go:414] "containerMap: removing container" podUID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" containerName="maas-api" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672558 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" containerName="maas-api" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672581 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672589 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672613 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bae87f4e-e00e-4e03-acb5-867a3cae39e2" containerName="payload-processing" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672622 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae87f4e-e00e-4e03-acb5-867a3cae39e2" containerName="payload-processing" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672633 5647 cpu_manager.go:414] "containerMap: removing container" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672640 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" Nov 02 17:55:49.672662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672657 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f189dc21-699c-4947-84c6-200e7f4adc4a" containerName="payload-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672664 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f189dc21-699c-4947-84c6-200e7f4adc4a" containerName="payload-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672682 5647 cpu_manager.go:414] "containerMap: removing container" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672690 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672713 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e7ea30ba-2a57-4111-ab0e-c8b9eb458785" containerName="maas-api" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672720 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ea30ba-2a57-4111-ab0e-c8b9eb458785" containerName="maas-api" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672755 5647 cpu_manager.go:414] "containerMap: removing container" podUID="49445ce7-b2d9-4714-b340-2e4426ac72f6" containerName="payload-pre-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672765 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49445ce7-b2d9-4714-b340-2e4426ac72f6" containerName="payload-pre-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672781 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" containerName="payload-pre-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672788 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" containerName="payload-pre-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672812 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672819 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672973 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="49445ce7-b2d9-4714-b340-2e4426ac72f6" containerName="payload-pre-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.672989 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4523c9b1-2b78-4b6f-9f06-e93a92f8fc27" containerName="payload-pre-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673002 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673014 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="010bf490-d048-49d4-b7c9-ac2dcd50c4f3" containerName="maas-api" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673026 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0753785-7233-4533-a398-0059cd0b8758" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673039 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae87f4e-e00e-4e03-acb5-867a3cae39e2" containerName="payload-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673049 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bdb11b-0842-4c0c-a3e6-8f61d3aa1de8" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673058 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f189dc21-699c-4947-84c6-200e7f4adc4a" containerName="payload-processing" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673068 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="76250067-6ed8-4158-8363-30bac127a046" containerName="istio-proxy" Nov 02 17:55:49.673100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.673077 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ea30ba-2a57-4111-ab0e-c8b9eb458785" containerName="maas-api" Nov 02 17:55:49.678177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.678157 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.694348 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.694125 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.694685 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.694575 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.694801 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.694896 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.694953 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsst6\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.695023 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.695054 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.695082 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.695314 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.695117 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.746757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.740169 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-943ca1-openshift-default-dockercfg-mcn9f\"" Nov 02 17:55:49.831244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.830988 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831077 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsst6\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831148 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831180 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831205 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831244 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831237 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831319 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.831776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.831762 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.838769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.832629 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.838769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.832766 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.838769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.833076 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.838769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.833701 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.838769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.834130 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.839937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.839833 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk"] Nov 02 17:55:49.860917 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.846006 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.869808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.869507 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.869808 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.869710 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.893635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.893221 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.896885 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.896806 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsst6\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6\") pod \"e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:49.988941 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.988841 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb"] Nov 02 17:55:49.989145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.989110 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" containerID="cri-o://873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741" gracePeriod=30 Nov 02 17:55:49.995578 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:49.995321 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:50.431021 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:50.430987 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk"] Nov 02 17:55:50.432139 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:55:50.432109 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06813a2_b541_443d_9415_ebcdc7a43438.slice/crio-45d1e5edd8458373907696d1b76218c430522ce0f6a0e181c4ffefceac962a9a WatchSource:0}: Error finding container 45d1e5edd8458373907696d1b76218c430522ce0f6a0e181c4ffefceac962a9a: Status 404 returned error can't find the container with id 45d1e5edd8458373907696d1b76218c430522ce0f6a0e181c4ffefceac962a9a Nov 02 17:55:50.434481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:50.434444 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:55:50.434561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:50.434516 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:55:50.434597 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:50.434564 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:55:51.351647 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:51.351607 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" event={"ID":"e06813a2-b541-443d-9415-ebcdc7a43438","Type":"ContainerStarted","Data":"f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746"} Nov 02 17:55:51.352197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:51.351661 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" event={"ID":"e06813a2-b541-443d-9415-ebcdc7a43438","Type":"ContainerStarted","Data":"45d1e5edd8458373907696d1b76218c430522ce0f6a0e181c4ffefceac962a9a"} Nov 02 17:55:51.834259 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:51.834141 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podStartSLOduration=2.83412461 podStartE2EDuration="2.83412461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:51.577918053 +0000 UTC m=+1938.651445257" watchObservedRunningTime="2026-11-02 17:55:51.83412461 +0000 UTC m=+1938.907651812" Nov 02 17:55:51.835891 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:51.835863 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:55:51.996695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:51.996661 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:52.003087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.003056 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:52.360099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.360065 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:52.363955 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.363925 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:55:52.819125 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.819099 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:55:52.848393 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848361 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.848663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848469 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.848663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848501 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.848663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848577 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.848663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848618 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.848663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848651 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.849104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848700 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.849104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848800 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.849104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848832 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert\") pod \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\" (UID: \"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9\") " Nov 02 17:55:52.849104 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.848938 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.849321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849156 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.849321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849171 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.849321 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849222 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.849500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849476 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:52.849816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849794 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.849872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849824 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.849872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849843 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.849872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849859 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.849963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.849876 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.850953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.850926 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:52.851076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.851051 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:52.851342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.851322 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:52.851422 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.851346 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92" pod "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" (UID: "383a1dc5-e7b5-43ba-a3e1-a5c9781584b9"). InnerVolumeSpecName "kube-api-access-2mz92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:52.951181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.951091 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mz92\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-kube-api-access-2mz92\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.951181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.951124 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.951181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.951134 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:52.951181 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:52.951143 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:53.368559 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.368524 5647 generic.go:356] "Generic (PLEG): container finished" podID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerID="56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f" exitCode=0 Nov 02 17:55:53.369012 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.368612 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" Nov 02 17:55:53.369012 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.368612 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" event={"ID":"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9","Type":"ContainerDied","Data":"56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f"} Nov 02 17:55:53.369012 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.368685 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q" event={"ID":"383a1dc5-e7b5-43ba-a3e1-a5c9781584b9","Type":"ContainerDied","Data":"fba86cec35bb675afe45603300943bbdb92cd4c9c6e2ebce8c9deaaf8759b465"} Nov 02 17:55:53.369012 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.368718 5647 scope.go:122] "RemoveContainer" containerID="56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f" Nov 02 17:55:53.379086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.379063 5647 scope.go:122] "RemoveContainer" containerID="109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115" Nov 02 17:55:53.388606 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.388521 5647 scope.go:122] "RemoveContainer" containerID="56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f" Nov 02 17:55:53.389057 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:53.389031 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f\": container with ID starting with 56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f not found: ID does not exist" containerID="56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f" Nov 02 17:55:53.389120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.389070 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f"} err="failed to get container status \"56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f\": rpc error: code = NotFound desc = could not find container \"56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f\": container with ID starting with 56639d0e2f10e617cc549608632e4063b6ca8b112f038f3f6b8415e46370ac3f not found: ID does not exist" Nov 02 17:55:53.389120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.389111 5647 scope.go:122] "RemoveContainer" containerID="109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115" Nov 02 17:55:53.389467 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:53.389442 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115\": container with ID starting with 109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115 not found: ID does not exist" containerID="109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115" Nov 02 17:55:53.389552 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.389477 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115"} err="failed to get container status \"109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115\": rpc error: code = NotFound desc = could not find container \"109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115\": container with ID starting with 109a91a486daaf286df4ee8fd1902b7006fe5be8438bcc64c5a7da3dd099e115 not found: ID does not exist" Nov 02 17:55:53.535681 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.535644 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q"] Nov 02 17:55:53.586514 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:53.586480 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-666de0-openshift-default-95564b6cf-hkx2q"] Nov 02 17:55:55.082815 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.082784 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" path="/var/lib/kubelet/pods/383a1dc5-e7b5-43ba-a3e1-a5c9781584b9/volumes" Nov 02 17:55:55.265234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.265207 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:55:55.384352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.384260 5647 generic.go:356] "Generic (PLEG): container finished" podID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerID="873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741" exitCode=0 Nov 02 17:55:55.384352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.384311 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" Nov 02 17:55:55.384352 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.384339 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" event={"ID":"3f284f18-defa-49a4-8b36-52ff0ef8eb13","Type":"ContainerDied","Data":"873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741"} Nov 02 17:55:55.384662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.384400 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb" event={"ID":"3f284f18-defa-49a4-8b36-52ff0ef8eb13","Type":"ContainerDied","Data":"b38995921d45533fdac9e4b860828340998ded07ccba3d1548d7699a6d0fbe92"} Nov 02 17:55:55.384662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.384452 5647 scope.go:122] "RemoveContainer" containerID="873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741" Nov 02 17:55:55.385280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385246 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385289 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385351 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385391 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385431 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385493 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385580 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385656 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385773 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:55.385900 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385804 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert\") pod \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\" (UID: \"3f284f18-defa-49a4-8b36-52ff0ef8eb13\") " Nov 02 17:55:55.385960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.385940 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:55.386366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.386316 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.386366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.386339 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.386497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.386398 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:55.386553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.386504 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:55.386676 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.386653 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:55:55.392177 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.392147 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "kube-api-access-nzdt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:55.392386 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.392362 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:55:55.392470 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.392453 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:55:55.393092 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.393067 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token" pod "3f284f18-defa-49a4-8b36-52ff0ef8eb13" (UID: "3f284f18-defa-49a4-8b36-52ff0ef8eb13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:55:55.415628 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.415604 5647 scope.go:122] "RemoveContainer" containerID="7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f" Nov 02 17:55:55.426003 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.425977 5647 scope.go:122] "RemoveContainer" containerID="873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741" Nov 02 17:55:55.426372 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:55.426343 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741\": container with ID starting with 873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741 not found: ID does not exist" containerID="873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741" Nov 02 17:55:55.426446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.426386 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741"} err="failed to get container status \"873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741\": rpc error: code = NotFound desc = could not find container \"873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741\": container with ID starting with 873a65109a54d005c29e897f0c471ba1c71899e541f23b55102062a75cd59741 not found: ID does not exist" Nov 02 17:55:55.426446 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.426413 5647 scope.go:122] "RemoveContainer" containerID="7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f" Nov 02 17:55:55.426715 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:55:55.426691 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f\": container with ID starting with 7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f not found: ID does not exist" containerID="7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f" Nov 02 17:55:55.427049 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.426723 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f"} err="failed to get container status \"7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f\": rpc error: code = NotFound desc = could not find container \"7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f\": container with ID starting with 7879045d7dd5dcdb7c11765213beaff70ace447c3736f0cfab38779d34df8c5f not found: ID does not exist" Nov 02 17:55:55.487198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487161 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.487198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487192 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.487198 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487202 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.487447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487211 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzdt2\" (UniqueName: \"kubernetes.io/projected/3f284f18-defa-49a4-8b36-52ff0ef8eb13-kube-api-access-nzdt2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.487447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487221 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.487447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487230 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.487447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.487238 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f284f18-defa-49a4-8b36-52ff0ef8eb13-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:55:55.825926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.825884 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb"] Nov 02 17:55:55.832700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:55.832669 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3fa615-openshift-default-85648c588c-6wlcb"] Nov 02 17:55:57.082270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.082236 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" path="/var/lib/kubelet/pods/3f284f18-defa-49a4-8b36-52ff0ef8eb13/volumes" Nov 02 17:55:57.969054 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969018 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l"] Nov 02 17:55:57.969581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969567 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" Nov 02 17:55:57.969639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969583 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" Nov 02 17:55:57.969639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969633 5647 cpu_manager.go:414] "containerMap: removing container" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" Nov 02 17:55:57.969639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969638 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" Nov 02 17:55:57.969769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969646 5647 cpu_manager.go:414] "containerMap: removing container" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" Nov 02 17:55:57.969769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969652 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" Nov 02 17:55:57.969769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969726 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" Nov 02 17:55:57.969769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969748 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" Nov 02 17:55:57.969769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969754 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="383a1dc5-e7b5-43ba-a3e1-a5c9781584b9" containerName="istio-proxy" Nov 02 17:55:57.969769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.969762 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" Nov 02 17:55:57.974330 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:57.974311 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.031385 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.031348 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l"] Nov 02 17:55:58.083759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.081587 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-943ca1-dockercfg-w4kkb\"" Nov 02 17:55:58.083759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.081666 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-943ca1\"" Nov 02 17:55:58.109757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.100399 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.109757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.100534 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcxkf\" (UniqueName: \"kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf\") pod \"payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.135757 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.135412 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9"] Nov 02 17:55:58.145425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.136364 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" Nov 02 17:55:58.145425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.136388 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f284f18-defa-49a4-8b36-52ff0ef8eb13" containerName="istio-proxy" Nov 02 17:55:58.179149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.179116 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.186886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.186852 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9"] Nov 02 17:55:58.201366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.201336 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.201601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.201408 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcxkf\" (UniqueName: \"kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf\") pod \"payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.201601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.201453 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.201601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.201479 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt69\" (UniqueName: \"kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69\") pod \"payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.202088 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.202068 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.260680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.260602 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcxkf\" (UniqueName: \"kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf\") pod \"payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.287893 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.287849 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:58.303162 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.303136 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.303301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.303168 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt69\" (UniqueName: \"kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69\") pod \"payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.303873 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.303848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.370418 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.370359 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt69\" (UniqueName: \"kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69\") pod \"payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.496558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.496521 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:58.667421 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.667391 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l"] Nov 02 17:55:58.669862 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:55:58.669831 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c37530_b9f9_432c_8115_1ab575ef0eef.slice/crio-709d908cd34bbc14b0b9bf65ca169a3444cad0b22c37f5514ecafe8a6b4a76df WatchSource:0}: Error finding container 709d908cd34bbc14b0b9bf65ca169a3444cad0b22c37f5514ecafe8a6b4a76df: Status 404 returned error can't find the container with id 709d908cd34bbc14b0b9bf65ca169a3444cad0b22c37f5514ecafe8a6b4a76df Nov 02 17:55:58.872496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:58.872287 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9"] Nov 02 17:55:58.874903 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:55:58.874876 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba1257_b531_4d87_b7f8_e2982a7b24e6.slice/crio-066da305b5e16dd091599cc2abd5ffc3bd447e6ce483afbdae29be82044cb04b WatchSource:0}: Error finding container 066da305b5e16dd091599cc2abd5ffc3bd447e6ce483afbdae29be82044cb04b: Status 404 returned error can't find the container with id 066da305b5e16dd091599cc2abd5ffc3bd447e6ce483afbdae29be82044cb04b Nov 02 17:55:59.415028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.414989 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" event={"ID":"66ba1257-b531-4d87-b7f8-e2982a7b24e6","Type":"ContainerStarted","Data":"9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a"} Nov 02 17:55:59.415528 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.415057 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" event={"ID":"66ba1257-b531-4d87-b7f8-e2982a7b24e6","Type":"ContainerStarted","Data":"066da305b5e16dd091599cc2abd5ffc3bd447e6ce483afbdae29be82044cb04b"} Nov 02 17:55:59.415528 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.415092 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:55:59.416799 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.416758 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" event={"ID":"78c37530-b9f9-432c-8115-1ab575ef0eef","Type":"ContainerStarted","Data":"f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826"} Nov 02 17:55:59.416936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.416821 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" event={"ID":"78c37530-b9f9-432c-8115-1ab575ef0eef","Type":"ContainerStarted","Data":"709d908cd34bbc14b0b9bf65ca169a3444cad0b22c37f5514ecafe8a6b4a76df"} Nov 02 17:55:59.416936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.416864 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:55:59.532864 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:55:59.532800 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" podStartSLOduration=1.532786372 podStartE2EDuration="1.532786372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:59.532105599 +0000 UTC m=+1946.605632815" watchObservedRunningTime="2026-11-02 17:55:59.532786372 +0000 UTC m=+1946.606313575" Nov 02 17:56:00.876252 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.876190 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" podStartSLOduration=3.8761730930000002 podStartE2EDuration="3.876173093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:59.660020112 +0000 UTC m=+1946.733547315" watchObservedRunningTime="2026-11-02 17:56:00.876173093 +0000 UTC m=+1947.949700306" Nov 02 17:56:00.878183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.878162 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f"] Nov 02 17:56:00.882475 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.882455 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:00.935706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.935674 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:00.935904 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.935775 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:00.935960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.935938 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:00.936011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.935995 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddkq\" (UniqueName: \"kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:00.938691 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.938666 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1cea7c-serving-cert\"" Nov 02 17:56:00.992004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:00.991961 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f"] Nov 02 17:56:01.037498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.037459 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.037675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.037552 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddkq\" (UniqueName: \"kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.037727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.037675 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.037727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.037708 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.037972 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:01.037916 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-serving-cert: secret "maas-api-e2e-api-b-1cea7c-serving-cert" not found Nov 02 17:56:01.038066 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:01.038001 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls podName:b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60 nodeName:}" failed. No retries permitted until 2026-11-02 17:56:01.53797879 +0000 UTC m=+1948.611505979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls") pod "maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" (UID: "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60") : secret "maas-api-e2e-api-b-1cea7c-serving-cert" not found Nov 02 17:56:01.038239 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.038215 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.040332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.040303 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.096880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.096840 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddkq\" (UniqueName: \"kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.548126 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.548085 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.550698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.550664 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\") pod \"maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:01.797508 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:01.797466 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:02.181499 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:02.181461 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f"] Nov 02 17:56:02.183387 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:02.183345 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41315a8_7dfc_4dfd_8fb7_0ee1cf38ae60.slice/crio-7a0312af9d51a65322780e7771c0224561d522a51d88a7109d1ceda57fe2311c WatchSource:0}: Error finding container 7a0312af9d51a65322780e7771c0224561d522a51d88a7109d1ceda57fe2311c: Status 404 returned error can't find the container with id 7a0312af9d51a65322780e7771c0224561d522a51d88a7109d1ceda57fe2311c Nov 02 17:56:02.436773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:02.436660 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" event={"ID":"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60","Type":"ContainerStarted","Data":"7a0312af9d51a65322780e7771c0224561d522a51d88a7109d1ceda57fe2311c"} Nov 02 17:56:03.444787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:03.444707 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" event={"ID":"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60","Type":"ContainerStarted","Data":"b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83"} Nov 02 17:56:03.444787 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:03.444793 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:03.564650 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:03.564593 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" podStartSLOduration=2.702091055 podStartE2EDuration="3.564573131s" totalImagesPullingTime="862.482076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:03.562825759 +0000 UTC m=+1950.636352975" watchObservedRunningTime="2026-11-02 17:56:03.564573131 +0000 UTC m=+1950.638100334" Nov 02 17:56:05.947863 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:05.947824 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:56:09.456439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:09.456410 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:56:10.426271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:10.426234 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 17:56:10.426425 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:10.426303 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 17:56:15.596919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:15.596883 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk"] Nov 02 17:56:15.597433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:15.597065 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" podUID="a3f8bb41-341d-47bd-96c2-d61e3710434f" containerName="maas-api" containerID="cri-o://73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd" gracePeriod=30 Nov 02 17:56:15.927846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:15.927824 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:56:16.059604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.059334 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca\") pod \"a3f8bb41-341d-47bd-96c2-d61e3710434f\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " Nov 02 17:56:16.059604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.059421 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs\") pod \"a3f8bb41-341d-47bd-96c2-d61e3710434f\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " Nov 02 17:56:16.059604 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.059452 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss\" (UniqueName: \"kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss\") pod \"a3f8bb41-341d-47bd-96c2-d61e3710434f\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " Nov 02 17:56:16.059985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.059609 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\") pod \"a3f8bb41-341d-47bd-96c2-d61e3710434f\" (UID: \"a3f8bb41-341d-47bd-96c2-d61e3710434f\") " Nov 02 17:56:16.060260 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.060226 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca" pod "a3f8bb41-341d-47bd-96c2-d61e3710434f" (UID: "a3f8bb41-341d-47bd-96c2-d61e3710434f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:16.063020 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.062973 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls" pod "a3f8bb41-341d-47bd-96c2-d61e3710434f" (UID: "a3f8bb41-341d-47bd-96c2-d61e3710434f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:16.063408 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.063379 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss" pod "a3f8bb41-341d-47bd-96c2-d61e3710434f" (UID: "a3f8bb41-341d-47bd-96c2-d61e3710434f"). InnerVolumeSpecName "kube-api-access-vmcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:16.063867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.063839 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs" pod "a3f8bb41-341d-47bd-96c2-d61e3710434f" (UID: "a3f8bb41-341d-47bd-96c2-d61e3710434f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:56:16.161931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.161841 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:16.161931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.161870 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmcss\" (UniqueName: \"kubernetes.io/projected/a3f8bb41-341d-47bd-96c2-d61e3710434f-kube-api-access-vmcss\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:16.161931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.161887 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3f8bb41-341d-47bd-96c2-d61e3710434f-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:16.161931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.161900 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f8bb41-341d-47bd-96c2-d61e3710434f-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:16.634318 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.634270 5647 generic.go:356] "Generic (PLEG): container finished" podID="a3f8bb41-341d-47bd-96c2-d61e3710434f" containerID="73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd" exitCode=0 Nov 02 17:56:16.635089 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.635041 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" event={"ID":"a3f8bb41-341d-47bd-96c2-d61e3710434f","Type":"ContainerDied","Data":"73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd"} Nov 02 17:56:16.635317 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.635280 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" event={"ID":"a3f8bb41-341d-47bd-96c2-d61e3710434f","Type":"ContainerDied","Data":"da4511a00747a96266ce56740f7d54f2db974187dc1994febd6e873519c92a62"} Nov 02 17:56:16.635468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.635449 5647 scope.go:122] "RemoveContainer" containerID="73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd" Nov 02 17:56:16.635880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.635855 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk" Nov 02 17:56:16.694245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.693374 5647 scope.go:122] "RemoveContainer" containerID="73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd" Nov 02 17:56:16.697298 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:16.697218 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd\": container with ID starting with 73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd not found: ID does not exist" containerID="73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd" Nov 02 17:56:16.697298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.697261 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd"} err="failed to get container status \"73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd\": rpc error: code = NotFound desc = could not find container \"73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd\": container with ID starting with 73419b641a37145bb3a32834b189cadf2a01208afd17284601aba2e99388d1cd not found: ID does not exist" Nov 02 17:56:16.837281 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.837249 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk"] Nov 02 17:56:16.887998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:16.887906 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f10385-64cf88b945-4mgdk"] Nov 02 17:56:17.082881 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.082851 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f8bb41-341d-47bd-96c2-d61e3710434f" path="/var/lib/kubelet/pods/a3f8bb41-341d-47bd-96c2-d61e3710434f/volumes" Nov 02 17:56:17.181835 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.181746 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx"] Nov 02 17:56:17.182417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.182392 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a3f8bb41-341d-47bd-96c2-d61e3710434f" containerName="maas-api" Nov 02 17:56:17.182417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.182419 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f8bb41-341d-47bd-96c2-d61e3710434f" containerName="maas-api" Nov 02 17:56:17.182623 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.182611 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f8bb41-341d-47bd-96c2-d61e3710434f" containerName="maas-api" Nov 02 17:56:17.188717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.188693 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.241982 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.241952 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-9ac699-serving-cert\"" Nov 02 17:56:17.243566 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.243531 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx"] Nov 02 17:56:17.302940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.302908 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.303142 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.303063 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7fc\" (UniqueName: \"kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.303142 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.303133 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.303931 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.303201 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.404160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.404122 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7fc\" (UniqueName: \"kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.404160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.404163 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.404419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.404209 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.404419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.404307 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.404419 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:17.404307 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-serving-cert: secret "maas-api-e2e-ipp-b-9ac699-serving-cert" not found Nov 02 17:56:17.404574 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:17.404445 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls podName:594e29d2-f6c2-4048-b33b-81f8a6c4b92c nodeName:}" failed. No retries permitted until 2026-11-02 17:56:17.904424371 +0000 UTC m=+1964.977951575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls") pod "maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" (UID: "594e29d2-f6c2-4048-b33b-81f8a6c4b92c") : secret "maas-api-e2e-ipp-b-9ac699-serving-cert" not found Nov 02 17:56:17.404923 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.404890 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.406880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.406858 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.486727 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.486354 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7fc\" (UniqueName: \"kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.912643 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.912602 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:17.915311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:17.915288 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:18.106091 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:18.106052 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:18.482547 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:18.482522 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx"] Nov 02 17:56:18.483557 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:18.483533 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594e29d2_f6c2_4048_b33b_81f8a6c4b92c.slice/crio-bbfb5fcc48e61338511fea6ad20b9595e0db7329c536acbc6844472a2d528870 WatchSource:0}: Error finding container bbfb5fcc48e61338511fea6ad20b9595e0db7329c536acbc6844472a2d528870: Status 404 returned error can't find the container with id bbfb5fcc48e61338511fea6ad20b9595e0db7329c536acbc6844472a2d528870 Nov 02 17:56:18.669433 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:18.669398 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" event={"ID":"594e29d2-f6c2-4048-b33b-81f8a6c4b92c","Type":"ContainerStarted","Data":"bbfb5fcc48e61338511fea6ad20b9595e0db7329c536acbc6844472a2d528870"} Nov 02 17:56:19.340983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:19.340938 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:56:19.676943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:19.676905 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" event={"ID":"594e29d2-f6c2-4048-b33b-81f8a6c4b92c","Type":"ContainerStarted","Data":"a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334"} Nov 02 17:56:19.677120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:19.676988 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:19.800018 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:19.799941 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" podStartSLOduration=1.996537396 podStartE2EDuration="2.79992293s" totalImagesPullingTime="803.385534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:19.798714063 +0000 UTC m=+1966.872241268" watchObservedRunningTime="2026-11-02 17:56:19.79992293 +0000 UTC m=+1966.873450132" Nov 02 17:56:22.636840 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:22.636793 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn"] Nov 02 17:56:22.637267 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:22.637015 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" containerID="cri-o://666b959e6e7d97bb35e05381e1f9ed68051f2e2397ae625bb78dda06169883ab" gracePeriod=30 Nov 02 17:56:24.553152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.553117 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9"] Nov 02 17:56:24.557546 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.557522 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.624079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.623093 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5a1e27-openshift-default-dockercfg-66hh4\"" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636489 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636552 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636580 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636608 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636656 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636773 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636807 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkmm\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.640772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.636868 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.753773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.752815 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9"] Nov 02 17:56:24.755814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.755621 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.770882 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.770941 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771005 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771110 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771143 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771217 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771260 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkmm\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771343 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.771805 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.757826 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772514 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.772436 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.772946 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.772914 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.773675 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.773658 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.793222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.778631 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.793222 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.783504 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.850790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.846517 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.880788 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.879952 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkmm\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm\") pod \"e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:24.926765 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:24.912056 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:25.365640 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.365609 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9"] Nov 02 17:56:25.366543 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:25.366517 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8aa7ff_0db9_4cea_a1a6_0b24469e7ad1.slice/crio-67511c6a458343354151b4c0a5c34f4afee92c4dc05a20984998a96faf4fce9a WatchSource:0}: Error finding container 67511c6a458343354151b4c0a5c34f4afee92c4dc05a20984998a96faf4fce9a: Status 404 returned error can't find the container with id 67511c6a458343354151b4c0a5c34f4afee92c4dc05a20984998a96faf4fce9a Nov 02 17:56:25.368986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.368952 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:25.369044 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.369016 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:25.369090 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.369045 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:25.689068 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.688981 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:56:25.847333 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.847291 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" event={"ID":"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1","Type":"ContainerStarted","Data":"fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc"} Nov 02 17:56:25.847523 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.847408 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" event={"ID":"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1","Type":"ContainerStarted","Data":"67511c6a458343354151b4c0a5c34f4afee92c4dc05a20984998a96faf4fce9a"} Nov 02 17:56:25.927763 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.912831 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:25.967107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:25.966987 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podStartSLOduration=1.966970296 podStartE2EDuration="1.966970296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:25.966679829 +0000 UTC m=+1973.040207036" watchObservedRunningTime="2026-11-02 17:56:25.966970296 +0000 UTC m=+1973.040497499" Nov 02 17:56:26.476078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:26.476039 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 17:56:26.476249 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:26.476122 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 17:56:26.918285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:26.918255 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:27.866431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:27.866396 5647 generic.go:356] "Generic (PLEG): container finished" podID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerID="666b959e6e7d97bb35e05381e1f9ed68051f2e2397ae625bb78dda06169883ab" exitCode=0 Nov 02 17:56:27.867574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:27.867544 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" event={"ID":"d01ea49e-000a-40ec-b40c-d461cc102fbf","Type":"ContainerDied","Data":"666b959e6e7d97bb35e05381e1f9ed68051f2e2397ae625bb78dda06169883ab"} Nov 02 17:56:27.867710 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:27.867610 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:27.867710 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:27.867643 5647 scope.go:122] "RemoveContainer" containerID="b940f52dd8abe7e55bc5453b4d82498a2165bbb72f2226655e5df951723273b5" Nov 02 17:56:27.869426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:27.868794 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:56:27.929179 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:27.929152 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:56:28.015155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015119 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015174 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015340 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015281 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015388 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015430 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015471 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:28.015589 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015526 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015701 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015591 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015701 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015646 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.015701 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015691 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo\") pod \"d01ea49e-000a-40ec-b40c-d461cc102fbf\" (UID: \"d01ea49e-000a-40ec-b40c-d461cc102fbf\") " Nov 02 17:56:28.016008 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015985 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:56:28.016204 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.016181 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:28.016280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.015912 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:28.016489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.016349 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.016489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.016372 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01ea49e-000a-40ec-b40c-d461cc102fbf-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.016489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.016386 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.016489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.016402 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.016489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.016462 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:28.018120 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.018075 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "kube-api-access-2pzcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:28.018645 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.018619 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:56:28.018725 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.018662 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:56:28.019706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.019682 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token" pod "d01ea49e-000a-40ec-b40c-d461cc102fbf" (UID: "d01ea49e-000a-40ec-b40c-d461cc102fbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:56:28.117563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.117530 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzcd\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-kube-api-access-2pzcd\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.117563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.117560 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.117563 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.117570 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.117818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.117579 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.117818 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.117590 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01ea49e-000a-40ec-b40c-d461cc102fbf-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:56:28.874390 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.874359 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" Nov 02 17:56:28.874587 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.874396 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn" event={"ID":"d01ea49e-000a-40ec-b40c-d461cc102fbf","Type":"ContainerDied","Data":"0e7c2fcfb279a68664aff7af8c5323e6a891532263b6836892c976e91a5764d6"} Nov 02 17:56:28.874587 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:28.874457 5647 scope.go:122] "RemoveContainer" containerID="666b959e6e7d97bb35e05381e1f9ed68051f2e2397ae625bb78dda06169883ab" Nov 02 17:56:29.095506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.095472 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn"] Nov 02 17:56:29.103287 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.103257 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f10385-openshift-default-8b4d9b495-lx2bn"] Nov 02 17:56:29.757553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.757514 5647 patch_prober.go:29] interesting pod/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": read tcp 10.134.0.2:43920->10.134.0.157:15021: read: connection reset by peer" start-of-body= Nov 02 17:56:29.757874 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.757584 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": read tcp 10.134.0.2:43920->10.134.0.157:15021: read: connection reset by peer" Nov 02 17:56:29.882468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.882435 5647 generic.go:356] "Generic (PLEG): container finished" podID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerID="612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1" exitCode=0 Nov 02 17:56:29.882652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.882509 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" event={"ID":"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20","Type":"ContainerDied","Data":"612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1"} Nov 02 17:56:29.883076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.883053 5647 scope.go:122] "RemoveContainer" containerID="612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1" Nov 02 17:56:29.884595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.884565 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:29.884662 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.884639 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:29.884700 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:29.884672 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:30.891934 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:30.891897 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" event={"ID":"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20","Type":"ContainerStarted","Data":"3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d"} Nov 02 17:56:31.081431 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:31.081399 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" path="/var/lib/kubelet/pods/d01ea49e-000a-40ec-b40c-d461cc102fbf/volumes" Nov 02 17:56:31.395174 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:31.395142 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:56:31.395419 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:31.395393 5647 patch_prober.go:29] interesting pod/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:56:31.395476 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:31.395456 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 17:56:32.396100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.396060 5647 patch_prober.go:29] interesting pod/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:56:32.396504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.396141 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 17:56:32.429821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.429783 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd"] Nov 02 17:56:32.430445 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.430430 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" Nov 02 17:56:32.430496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.430446 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" Nov 02 17:56:32.430586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.430577 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" Nov 02 17:56:32.430639 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.430600 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" Nov 02 17:56:32.433394 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.433372 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.477776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.477713 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd"] Nov 02 17:56:32.489560 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.489514 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9js5j\" (UniqueName: \"kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j\") pod \"payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.490311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.490278 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.522819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.518030 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5a1e27\"" Nov 02 17:56:32.559760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.553586 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5a1e27-dockercfg-prbxf\"" Nov 02 17:56:32.645759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.608701 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.665376 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.652081 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9js5j\" (UniqueName: \"kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j\") pod \"payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.665833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.659371 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.733565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.733530 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9js5j\" (UniqueName: \"kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j\") pod \"payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:32.745908 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:32.745880 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:33.159378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:33.159327 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd"] Nov 02 17:56:33.161054 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:33.161013 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3fab43_cd7c_486d_bc78_6c1616a78a22.slice/crio-7272990bfc1182334638ff3a6319a0b02d019ff23219e97076dfb4f819968c25 WatchSource:0}: Error finding container 7272990bfc1182334638ff3a6319a0b02d019ff23219e97076dfb4f819968c25: Status 404 returned error can't find the container with id 7272990bfc1182334638ff3a6319a0b02d019ff23219e97076dfb4f819968c25 Nov 02 17:56:33.396150 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:33.396111 5647 patch_prober.go:29] interesting pod/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:56:33.396535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:33.396178 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 17:56:33.947924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:33.947882 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" event={"ID":"da3fab43-cd7c-486d-bc78-6c1616a78a22","Type":"ContainerStarted","Data":"f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67"} Nov 02 17:56:33.948160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:33.947969 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" event={"ID":"da3fab43-cd7c-486d-bc78-6c1616a78a22","Type":"ContainerStarted","Data":"7272990bfc1182334638ff3a6319a0b02d019ff23219e97076dfb4f819968c25"} Nov 02 17:56:33.948160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:33.948128 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:34.065429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:34.065373 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" podStartSLOduration=2.065358489 podStartE2EDuration="2.065358489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:34.063378324 +0000 UTC m=+1981.136905527" watchObservedRunningTime="2026-11-02 17:56:34.065358489 +0000 UTC m=+1981.138885691" Nov 02 17:56:34.395945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:34.395907 5647 patch_prober.go:29] interesting pod/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:56:34.396230 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:34.396196 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 17:56:35.395661 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:35.395627 5647 patch_prober.go:29] interesting pod/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 02 17:56:35.395879 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:35.395698 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 02 17:56:35.958040 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:35.958007 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:56:36.400058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:36.400025 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:56:36.400355 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:36.400334 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:56:36.401123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:36.401106 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:56:37.539019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.538989 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7"] Nov 02 17:56:37.540185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.540156 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" Nov 02 17:56:37.540185 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.540185 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01ea49e-000a-40ec-b40c-d461cc102fbf" containerName="istio-proxy" Nov 02 17:56:37.545484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.545456 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571305 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571276 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571338 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571455 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571429 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571488 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571525 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571654 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571624 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zcjc\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571745 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571714 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571767 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.571868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.571856 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.599651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.599616 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d5924c-openshift-default-dockercfg-nk45c\"" Nov 02 17:56:37.653652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.653616 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7"] Nov 02 17:56:37.673136 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673096 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673199 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673289 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673339 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673533 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673371 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673601 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673535 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673672 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673591 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673840 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673819 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673890 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.673948 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.673892 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zcjc\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.674056 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.674015 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.674111 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.674065 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.674473 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.674442 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.674824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.674799 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.675925 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.675903 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.676718 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.676690 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.745468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.745432 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zcjc\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.746927 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.746898 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token\") pod \"e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:37.863484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:37.863449 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:38.304686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:38.304655 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7"] Nov 02 17:56:38.306242 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:38.306216 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653639db_490a_41c7_9ef1_a067d45c5f7b.slice/crio-8c9a7fff6d591797296165e84d831edb35330fcf4129928427b56de26b92e010 WatchSource:0}: Error finding container 8c9a7fff6d591797296165e84d831edb35330fcf4129928427b56de26b92e010: Status 404 returned error can't find the container with id 8c9a7fff6d591797296165e84d831edb35330fcf4129928427b56de26b92e010 Nov 02 17:56:38.308781 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:38.308708 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:38.308894 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:38.308827 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:38.308894 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:38.308879 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:38.999149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:38.999106 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" event={"ID":"653639db-490a-41c7-9ef1-a067d45c5f7b","Type":"ContainerStarted","Data":"31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f"} Nov 02 17:56:38.999794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:38.999182 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" event={"ID":"653639db-490a-41c7-9ef1-a067d45c5f7b","Type":"ContainerStarted","Data":"8c9a7fff6d591797296165e84d831edb35330fcf4129928427b56de26b92e010"} Nov 02 17:56:39.132884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:39.132829 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podStartSLOduration=2.132813611 podStartE2EDuration="2.132813611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:39.130959291 +0000 UTC m=+1986.204486494" watchObservedRunningTime="2026-11-02 17:56:39.132813611 +0000 UTC m=+1986.206340810" Nov 02 17:56:39.864332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:39.864293 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:39.870253 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:39.870226 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:40.028899 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:40.028857 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:40.029969 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:40.029944 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:56:44.959051 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:44.959019 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 17:56:47.156366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.156320 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk"] Nov 02 17:56:47.180108 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.180074 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.210332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.210298 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk"] Nov 02 17:56:47.222995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.211233 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.225228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.225199 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsm6q\" (UniqueName: \"kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q\") pod \"payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.253235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.252957 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d5924c-dockercfg-4hdmz\"" Nov 02 17:56:47.253235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.253065 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d5924c\"" Nov 02 17:56:47.303571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.303531 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89"] Nov 02 17:56:47.307620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.307590 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.326461 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.326427 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc56d\" (UniqueName: \"kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.326616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.326523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.326616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.326554 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsm6q\" (UniqueName: \"kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q\") pod \"payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.326858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.326833 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.327463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.327437 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.367103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.367071 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89"] Nov 02 17:56:47.385085 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.385052 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsm6q\" (UniqueName: \"kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q\") pod \"payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.428123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.428037 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc56d\" (UniqueName: \"kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.428311 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.428290 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.429034 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.429015 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.485992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.485958 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc56d\" (UniqueName: \"kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d\") pod \"payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.553819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.553785 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:47.623074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.622860 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:47.948782 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:47.948748 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk"] Nov 02 17:56:47.950945 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:47.950916 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3093163_fb0a_460b_881a_f716151ae3f8.slice/crio-1051ce4ab1cd6714de3a0efb0f8fb41d7f1c8c9fd00d7b755e2f1b7e5ae9f18c WatchSource:0}: Error finding container 1051ce4ab1cd6714de3a0efb0f8fb41d7f1c8c9fd00d7b755e2f1b7e5ae9f18c: Status 404 returned error can't find the container with id 1051ce4ab1cd6714de3a0efb0f8fb41d7f1c8c9fd00d7b755e2f1b7e5ae9f18c Nov 02 17:56:48.058215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:48.058186 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89"] Nov 02 17:56:48.059718 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:48.059685 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848412c4_4841_4fd2_ac35_a8d1c4a3b5c8.slice/crio-52b2ec70de14701d8dc70b98352036039c7b2b67e79dbd7e07c395edcab41ef4 WatchSource:0}: Error finding container 52b2ec70de14701d8dc70b98352036039c7b2b67e79dbd7e07c395edcab41ef4: Status 404 returned error can't find the container with id 52b2ec70de14701d8dc70b98352036039c7b2b67e79dbd7e07c395edcab41ef4 Nov 02 17:56:48.082610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:48.082578 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" event={"ID":"f3093163-fb0a-460b-881a-f716151ae3f8","Type":"ContainerStarted","Data":"7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5"} Nov 02 17:56:48.082754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:48.082639 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" event={"ID":"f3093163-fb0a-460b-881a-f716151ae3f8","Type":"ContainerStarted","Data":"1051ce4ab1cd6714de3a0efb0f8fb41d7f1c8c9fd00d7b755e2f1b7e5ae9f18c"} Nov 02 17:56:48.082754 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:48.082691 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:48.084219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:48.084187 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" event={"ID":"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8","Type":"ContainerStarted","Data":"52b2ec70de14701d8dc70b98352036039c7b2b67e79dbd7e07c395edcab41ef4"} Nov 02 17:56:48.202481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:48.202428 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" podStartSLOduration=1.202412968 podStartE2EDuration="1.202412968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:48.200898259 +0000 UTC m=+1995.274425462" watchObservedRunningTime="2026-11-02 17:56:48.202412968 +0000 UTC m=+1995.275940171" Nov 02 17:56:49.091492 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.091463 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" event={"ID":"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8","Type":"ContainerStarted","Data":"253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1"} Nov 02 17:56:49.091780 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.091532 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:56:49.093182 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.093159 5647 generic.go:356] "Generic (PLEG): container finished" podID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerID="2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e" exitCode=0 Nov 02 17:56:49.093266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.093228 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" event={"ID":"ff3cf331-f2be-42c5-8866-f9cf477086f5","Type":"ContainerDied","Data":"2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e"} Nov 02 17:56:49.093652 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.093637 5647 scope.go:122] "RemoveContainer" containerID="2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e" Nov 02 17:56:49.094990 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.094963 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:49.095060 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.095046 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:49.095098 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.095082 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:56:49.208844 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:49.208785 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" podStartSLOduration=2.208770602 podStartE2EDuration="2.208770602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:49.206953636 +0000 UTC m=+1996.280480884" watchObservedRunningTime="2026-11-02 17:56:49.208770602 +0000 UTC m=+1996.282297804" Nov 02 17:56:50.100580 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:50.100542 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" event={"ID":"ff3cf331-f2be-42c5-8866-f9cf477086f5","Type":"ContainerStarted","Data":"d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f"} Nov 02 17:56:50.288184 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:50.288151 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:56:50.288699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:50.288320 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:56:50.288699 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:50.288378 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 17:56:51.287428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:51.287393 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:56:51.287614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:51.287459 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 17:56:52.287868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:52.287826 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:56:52.288315 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:52.287898 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 17:56:53.288169 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:53.288131 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:56:53.288586 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:53.288192 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 17:56:54.287847 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:54.287810 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 02 17:56:54.288062 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:54.287865 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 02 17:56:55.323183 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:55.323100 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:56:55.323978 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:55.323957 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:56:55.324769 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:55.324719 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:56:56.120071 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.120036 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l"] Nov 02 17:56:56.124211 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.124194 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.164500 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.164455 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.164671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.164586 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhz8\" (UniqueName: \"kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.164801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.164678 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.164974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.164955 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.179273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.179247 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-943ca1-serving-cert\"" Nov 02 17:56:56.182430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.182402 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l"] Nov 02 17:56:56.266571 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.266530 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.266772 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.266644 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.266833 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.266787 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhz8\" (UniqueName: \"kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.266957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.266938 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.267122 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:56.267105 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-serving-cert: secret "maas-api-e2e-praxis-a-943ca1-serving-cert" not found Nov 02 17:56:56.267208 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:56:56.267174 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls podName:1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7 nodeName:}" failed. No retries permitted until 2026-11-02 17:56:56.767153656 +0000 UTC m=+2003.840680841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls") pod "maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" (UID: "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7") : secret "maas-api-e2e-praxis-a-943ca1-serving-cert" not found Nov 02 17:56:56.267423 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.267397 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.270014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.269986 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.325663 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.325634 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhz8\" (UniqueName: \"kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.774469 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.774431 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:56.777618 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:56.777589 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\") pod \"maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:57.038502 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:57.038406 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:57.416937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:57.416912 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l"] Nov 02 17:56:57.418557 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:56:57.418530 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee2ec0e_da4d_4b0d_ae19_812ddf147cc7.slice/crio-b6cd3a9a08b2666d17b6ead34ff3801d3f201eda7b2f1671fe6284e3042e2e4c WatchSource:0}: Error finding container b6cd3a9a08b2666d17b6ead34ff3801d3f201eda7b2f1671fe6284e3042e2e4c: Status 404 returned error can't find the container with id b6cd3a9a08b2666d17b6ead34ff3801d3f201eda7b2f1671fe6284e3042e2e4c Nov 02 17:56:58.178346 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:58.178308 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" event={"ID":"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7","Type":"ContainerStarted","Data":"b6cd3a9a08b2666d17b6ead34ff3801d3f201eda7b2f1671fe6284e3042e2e4c"} Nov 02 17:56:59.107420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:59.107384 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:56:59.187429 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:59.187398 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" event={"ID":"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7","Type":"ContainerStarted","Data":"2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66"} Nov 02 17:56:59.187631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:59.187539 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:56:59.341819 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:56:59.341753 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" podStartSLOduration=2.439047098 podStartE2EDuration="3.341726458s" totalImagesPullingTime="902.67936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:59.33966801 +0000 UTC m=+2006.413195236" watchObservedRunningTime="2026-11-02 17:56:59.341726458 +0000 UTC m=+2006.415253661" Nov 02 17:57:00.104000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:00.103971 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:57:02.682581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:02.682540 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:57:05.200678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:05.200636 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 17:57:05.948853 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:57:05.948807 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:57:21.481454 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:21.481405 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv"] Nov 02 17:57:26.482557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.482224 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" podUID="9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" containerName="payload-processing" containerID="cri-o://dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd" gracePeriod=25 Nov 02 17:57:26.758335 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.758306 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:57:26.877132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.877087 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn\" (UniqueName: \"kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn\") pod \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " Nov 02 17:57:26.877420 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.877148 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume\") pod \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\" (UID: \"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6\") " Nov 02 17:57:26.878144 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.878112 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume" pod "9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" (UID: "9e2f65b6-8e4f-419b-87a3-ab40983e8ec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:26.880323 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.880280 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn" pod "9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" (UID: "9e2f65b6-8e4f-419b-87a3-ab40983e8ec6"). InnerVolumeSpecName "kube-api-access-9zfcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:26.979167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.979132 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zfcn\" (UniqueName: \"kubernetes.io/projected/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-kube-api-access-9zfcn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:57:26.979167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:26.979163 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:57:27.391507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.391411 5647 generic.go:356] "Generic (PLEG): container finished" podID="9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" containerID="dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd" exitCode=0 Nov 02 17:57:27.391507 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.391465 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" Nov 02 17:57:27.391713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.391499 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" event={"ID":"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6","Type":"ContainerDied","Data":"dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd"} Nov 02 17:57:27.391713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.391560 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv" event={"ID":"9e2f65b6-8e4f-419b-87a3-ab40983e8ec6","Type":"ContainerDied","Data":"d3779b07ff2d79f4c6487c6d3b5385d32bcc5f2b81c9b171e99bff812efdc09d"} Nov 02 17:57:27.391713 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.391582 5647 scope.go:122] "RemoveContainer" containerID="dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd" Nov 02 17:57:27.402304 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.402276 5647 scope.go:122] "RemoveContainer" containerID="dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd" Nov 02 17:57:27.402667 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:57:27.402642 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd\": container with ID starting with dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd not found: ID does not exist" containerID="dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd" Nov 02 17:57:27.402747 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.402675 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd"} err="failed to get container status \"dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd\": rpc error: code = NotFound desc = could not find container \"dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd\": container with ID starting with dd3da7f16142e3a06be76805b37b9c5c6827196d0018c89e295584b9508665bd not found: ID does not exist" Nov 02 17:57:27.564886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.564848 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv"] Nov 02 17:57:27.614773 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:27.614717 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5bf666-69465559c-bsfgv"] Nov 02 17:57:29.100551 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:29.100507 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" path="/var/lib/kubelet/pods/9e2f65b6-8e4f-419b-87a3-ab40983e8ec6/volumes" Nov 02 17:57:30.413271 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:30.413239 5647 generic.go:356] "Generic (PLEG): container finished" podID="e06813a2-b541-443d-9415-ebcdc7a43438" containerID="f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746" exitCode=0 Nov 02 17:57:30.413714 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:30.413319 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" event={"ID":"e06813a2-b541-443d-9415-ebcdc7a43438","Type":"ContainerDied","Data":"f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746"} Nov 02 17:57:30.413798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:30.413787 5647 scope.go:122] "RemoveContainer" containerID="f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746" Nov 02 17:57:30.415309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:30.415279 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:57:30.415373 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:30.415348 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:57:30.415409 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:30.415380 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:57:31.432059 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:31.432025 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" event={"ID":"e06813a2-b541-443d-9415-ebcdc7a43438","Type":"ContainerStarted","Data":"2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2"} Nov 02 17:57:31.996760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:31.996321 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:57:31.996760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:31.996479 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:57:31.996760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:31.996535 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 17:57:32.996449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:32.996409 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:57:32.996997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:32.996484 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 17:57:33.996218 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:33.996179 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:57:33.996538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:33.996245 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 17:57:34.783747 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:34.783705 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:57:34.997074 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:34.997034 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:57:34.997514 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:34.997117 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 17:57:36.003896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:36.003518 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 02 17:57:36.003896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:36.003646 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 02 17:57:36.051623 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:57:36.051579 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:57:37.100989 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:37.100958 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:57:37.101520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:37.101472 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:57:37.102641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:37.102621 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 17:57:38.887790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.887747 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp"] Nov 02 17:57:38.888566 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.888541 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" containerName="payload-processing" Nov 02 17:57:38.888671 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.888567 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" containerName="payload-processing" Nov 02 17:57:38.888816 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.888801 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2f65b6-8e4f-419b-87a3-ab40983e8ec6" containerName="payload-processing" Nov 02 17:57:38.893638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.893617 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:38.936191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.936162 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp"] Nov 02 17:57:38.946794 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:38.946762 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5a1e27-serving-cert\"" Nov 02 17:57:39.022667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.022624 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.022667 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.022668 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjqv\" (UniqueName: \"kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.022973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.022794 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.023049 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.023031 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.156814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.155716 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.156814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.156016 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.156814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.156159 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjqv\" (UniqueName: \"kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.156814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.156266 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.160766 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:57:39.157702 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-serving-cert: secret "maas-api-e2e-praxis-a-5a1e27-serving-cert" not found Nov 02 17:57:39.160766 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:57:39.157814 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls podName:c989b4af-ae4f-4bc1-a8e2-8753a19cb76e nodeName:}" failed. No retries permitted until 2026-11-02 17:57:39.65779103 +0000 UTC m=+2046.731318215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls") pod "maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" (UID: "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e") : secret "maas-api-e2e-praxis-a-5a1e27-serving-cert" not found Nov 02 17:57:39.166574 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.166536 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.180901 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.175695 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.232084 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.232045 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjqv\" (UniqueName: \"kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.663911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.663872 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.666424 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.666403 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:39.807126 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:39.807095 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:40.184484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:40.184448 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp"] Nov 02 17:57:40.186323 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:57:40.186291 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc989b4af_ae4f_4bc1_a8e2_8753a19cb76e.slice/crio-4710f408f630c4c61ac4d685ee0f39a2695824391f5d5ca64909e297919b61d3 WatchSource:0}: Error finding container 4710f408f630c4c61ac4d685ee0f39a2695824391f5d5ca64909e297919b61d3: Status 404 returned error can't find the container with id 4710f408f630c4c61ac4d685ee0f39a2695824391f5d5ca64909e297919b61d3 Nov 02 17:57:40.188284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:40.188247 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 17:57:40.499452 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:40.499357 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" event={"ID":"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e","Type":"ContainerStarted","Data":"4710f408f630c4c61ac4d685ee0f39a2695824391f5d5ca64909e297919b61d3"} Nov 02 17:57:41.507504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:41.507466 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" event={"ID":"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e","Type":"ContainerStarted","Data":"29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4"} Nov 02 17:57:41.507957 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:41.507548 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:41.628626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:41.628562 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" podStartSLOduration=2.7666697080000002 podStartE2EDuration="3.628548029s" totalImagesPullingTime="861.878321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:41.625881669 +0000 UTC m=+2048.699408874" watchObservedRunningTime="2026-11-02 17:57:41.628548029 +0000 UTC m=+2048.702075232" Nov 02 17:57:47.519172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:47.519140 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 17:57:59.246118 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.246078 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd"] Nov 02 17:57:59.246630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.246331 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" podUID="94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" containerName="maas-api" containerID="cri-o://15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521" gracePeriod=30 Nov 02 17:57:59.632869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.632832 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:57:59.701809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.694477 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca\") pod \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " Nov 02 17:57:59.701809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.694530 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\") pod \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " Nov 02 17:57:59.701809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.694562 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g\" (UniqueName: \"kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g\") pod \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " Nov 02 17:57:59.701809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.694617 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs\") pod \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\" (UID: \"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c\") " Nov 02 17:57:59.701809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.700941 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca" pod "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" (UID: "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:57:59.701809 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.701590 5647 generic.go:356] "Generic (PLEG): container finished" podID="94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" containerID="15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521" exitCode=0 Nov 02 17:57:59.702280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.701885 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" event={"ID":"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c","Type":"ContainerDied","Data":"15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521"} Nov 02 17:57:59.702280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.701950 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" event={"ID":"94fa3f47-4c8c-4fb6-a903-85502bb1ff4c","Type":"ContainerDied","Data":"f40d12d533d0b24fb90d0c6c6965be80391cdf64e4512d88ba7c378a6e112441"} Nov 02 17:57:59.702280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.701985 5647 scope.go:122] "RemoveContainer" containerID="15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521" Nov 02 17:57:59.702280 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.702178 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd" Nov 02 17:57:59.778803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.757881 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs" pod "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" (UID: "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:59.778803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.759865 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls" pod "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" (UID: "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:57:59.778803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.761329 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t"] Nov 02 17:57:59.778803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.761651 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" podUID="9491ac0a-2ae5-48cc-85a1-5856520fb4d5" containerName="payload-processing" containerID="cri-o://ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07" gracePeriod=30 Nov 02 17:57:59.778803 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.767965 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g" pod "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" (UID: "94fa3f47-4c8c-4fb6-a903-85502bb1ff4c"). InnerVolumeSpecName "kube-api-access-w7b8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:57:59.789625 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.789584 5647 scope.go:122] "RemoveContainer" containerID="15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521" Nov 02 17:57:59.790221 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:57:59.790057 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521\": container with ID starting with 15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521 not found: ID does not exist" containerID="15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521" Nov 02 17:57:59.790221 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.790097 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521"} err="failed to get container status \"15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521\": rpc error: code = NotFound desc = could not find container \"15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521\": container with ID starting with 15f7d53bde5b1be494c7622d15226b7b47751465f751bb5a3d8ab5fd579c5521 not found: ID does not exist" Nov 02 17:57:59.797383 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.797350 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:57:59.797496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.797388 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:57:59.797496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.797403 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7b8g\" (UniqueName: \"kubernetes.io/projected/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-kube-api-access-w7b8g\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:57:59.797496 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.797418 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:57:59.837342 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.837305 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89"] Nov 02 17:57:59.837548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:57:59.837529 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" podUID="1f35f43b-a6e2-4870-b563-bbbd8202df8d" containerName="payload-pre-processing" containerID="cri-o://04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876" gracePeriod=30 Nov 02 17:58:00.145632 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.145602 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:58:00.189294 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.189269 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:58:00.236856 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.236813 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd"] Nov 02 17:58:00.240852 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.240813 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-700abe-84cb5648f6-lzckd"] Nov 02 17:58:00.337762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.335251 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss\" (UniqueName: \"kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss\") pod \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " Nov 02 17:58:00.337762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.335352 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume\") pod \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\" (UID: \"1f35f43b-a6e2-4870-b563-bbbd8202df8d\") " Nov 02 17:58:00.337762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.335415 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc\" (UniqueName: \"kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc\") pod \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " Nov 02 17:58:00.337762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.335513 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume\") pod \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\" (UID: \"9491ac0a-2ae5-48cc-85a1-5856520fb4d5\") " Nov 02 17:58:00.337762 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.337138 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume" pod "1f35f43b-a6e2-4870-b563-bbbd8202df8d" (UID: "1f35f43b-a6e2-4870-b563-bbbd8202df8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:00.398020 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.397931 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss" pod "1f35f43b-a6e2-4870-b563-bbbd8202df8d" (UID: "1f35f43b-a6e2-4870-b563-bbbd8202df8d"). InnerVolumeSpecName "kube-api-access-f6xss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:00.442131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.442088 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc" pod "9491ac0a-2ae5-48cc-85a1-5856520fb4d5" (UID: "9491ac0a-2ae5-48cc-85a1-5856520fb4d5"). InnerVolumeSpecName "kube-api-access-kxzvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:00.444648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.444610 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xss\" (UniqueName: \"kubernetes.io/projected/1f35f43b-a6e2-4870-b563-bbbd8202df8d-kube-api-access-f6xss\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:00.444648 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.444648 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f35f43b-a6e2-4870-b563-bbbd8202df8d-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:00.444869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.444665 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzvc\" (UniqueName: \"kubernetes.io/projected/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-kube-api-access-kxzvc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:00.446150 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.446122 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume" pod "9491ac0a-2ae5-48cc-85a1-5856520fb4d5" (UID: "9491ac0a-2ae5-48cc-85a1-5856520fb4d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:00.546522 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.546480 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9491ac0a-2ae5-48cc-85a1-5856520fb4d5-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:00.710290 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.710198 5647 generic.go:356] "Generic (PLEG): container finished" podID="9491ac0a-2ae5-48cc-85a1-5856520fb4d5" containerID="ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07" exitCode=0 Nov 02 17:58:00.710464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.710287 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" Nov 02 17:58:00.710464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.710285 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" event={"ID":"9491ac0a-2ae5-48cc-85a1-5856520fb4d5","Type":"ContainerDied","Data":"ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07"} Nov 02 17:58:00.710464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.710391 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t" event={"ID":"9491ac0a-2ae5-48cc-85a1-5856520fb4d5","Type":"ContainerDied","Data":"6304799334bdfdd8c5378535b8596c454d157d9d4898df30ed935f5d8cfd9fe0"} Nov 02 17:58:00.710464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.710427 5647 scope.go:122] "RemoveContainer" containerID="ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07" Nov 02 17:58:00.713033 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.713003 5647 generic.go:356] "Generic (PLEG): container finished" podID="1f35f43b-a6e2-4870-b563-bbbd8202df8d" containerID="04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876" exitCode=0 Nov 02 17:58:00.713165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.713053 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" event={"ID":"1f35f43b-a6e2-4870-b563-bbbd8202df8d","Type":"ContainerDied","Data":"04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876"} Nov 02 17:58:00.713165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.713098 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" Nov 02 17:58:00.713165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.713131 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89" event={"ID":"1f35f43b-a6e2-4870-b563-bbbd8202df8d","Type":"ContainerDied","Data":"b5ff6bb1781ed1259b64a4a52f61419cd349b37ae11fb5b7d13d4a449ee6fa33"} Nov 02 17:58:00.720692 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.720675 5647 scope.go:122] "RemoveContainer" containerID="ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07" Nov 02 17:58:00.721092 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:00.721060 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07\": container with ID starting with ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07 not found: ID does not exist" containerID="ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07" Nov 02 17:58:00.721219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.721104 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07"} err="failed to get container status \"ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07\": rpc error: code = NotFound desc = could not find container \"ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07\": container with ID starting with ecdc28da7c14e7718850f8c00dc3582b683fa248ba6004928c4221e9abb55f07 not found: ID does not exist" Nov 02 17:58:00.721219 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.721131 5647 scope.go:122] "RemoveContainer" containerID="04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876" Nov 02 17:58:00.732145 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.732122 5647 scope.go:122] "RemoveContainer" containerID="04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876" Nov 02 17:58:00.732477 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:00.732457 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876\": container with ID starting with 04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876 not found: ID does not exist" containerID="04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876" Nov 02 17:58:00.732537 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.732490 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876"} err="failed to get container status \"04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876\": rpc error: code = NotFound desc = could not find container \"04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876\": container with ID starting with 04b4d35df5af65605f13447d8170c86001b8c1f793bd2f75c284265d2c090876 not found: ID does not exist" Nov 02 17:58:00.881610 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.881578 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t"] Nov 02 17:58:00.885504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:00.885476 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-700abe-8489bc99b-qct6t"] Nov 02 17:58:01.059626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:01.054782 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89"] Nov 02 17:58:01.059626 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:01.059278 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-700abe-69b8b478f9-wdn89"] Nov 02 17:58:01.097635 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:01.097600 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f35f43b-a6e2-4870-b563-bbbd8202df8d" path="/var/lib/kubelet/pods/1f35f43b-a6e2-4870-b563-bbbd8202df8d/volumes" Nov 02 17:58:01.098248 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:01.098229 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9491ac0a-2ae5-48cc-85a1-5856520fb4d5" path="/var/lib/kubelet/pods/9491ac0a-2ae5-48cc-85a1-5856520fb4d5/volumes" Nov 02 17:58:01.098798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:01.098778 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" path="/var/lib/kubelet/pods/94fa3f47-4c8c-4fb6-a903-85502bb1ff4c/volumes" Nov 02 17:58:03.060868 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:03.060835 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9"] Nov 02 17:58:03.061301 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:03.061072 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" containerID="cri-o://0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b" gracePeriod=30 Nov 02 17:58:04.438505 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.438476 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf"] Nov 02 17:58:04.745504 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.745423 5647 generic.go:356] "Generic (PLEG): container finished" podID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerID="fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc" exitCode=0 Nov 02 17:58:04.745659 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.745498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" event={"ID":"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1","Type":"ContainerDied","Data":"fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc"} Nov 02 17:58:04.745703 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.745657 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" containerID="cri-o://7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a" gracePeriod=30 Nov 02 17:58:04.746350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.746001 5647 scope.go:122] "RemoveContainer" containerID="fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc" Nov 02 17:58:04.747896 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.747859 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:04.747985 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.747951 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:04.748026 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:04.747985 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:05.755263 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:05.755224 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" event={"ID":"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1","Type":"ContainerStarted","Data":"5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90"} Nov 02 17:58:05.926760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:05.924779 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:58:05.926760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:05.924931 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:58:05.926760 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:05.924987 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 02 17:58:05.960318 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:05.960278 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:58:06.927247 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:06.927207 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:58:06.927827 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:06.927292 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 02 17:58:07.916178 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:07.916136 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:58:07.916506 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:07.916473 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 02 17:58:08.378920 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.378894 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:58:08.479172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479093 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479189 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479210 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479247 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479347 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479283 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479354 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479381 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479407 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479451 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps\") pod \"9e00115b-49bd-4a20-b83b-c845f2b06343\" (UID: \"9e00115b-49bd-4a20-b83b-c845f2b06343\") " Nov 02 17:58:08.479594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479500 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:08.479594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479562 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:08.480031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479696 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:08.480031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479761 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:08.480031 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.479772 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:08.480378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.480061 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.480378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.480079 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e00115b-49bd-4a20-b83b-c845f2b06343-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.480378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.480088 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.480378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.480097 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.480378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.480105 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.481649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.481620 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:08.481851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.481831 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:08.481953 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.481933 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "kube-api-access-zshps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:08.482138 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.482116 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy" pod "9e00115b-49bd-4a20-b83b-c845f2b06343" (UID: "9e00115b-49bd-4a20-b83b-c845f2b06343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:08.581345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.581312 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.581345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.581340 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.581345 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.581352 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zshps\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-kube-api-access-zshps\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.581599 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.581362 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e00115b-49bd-4a20-b83b-c845f2b06343-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:08.778555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.778466 5647 generic.go:356] "Generic (PLEG): container finished" podID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerID="0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b" exitCode=0 Nov 02 17:58:08.778555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.778515 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" Nov 02 17:58:08.778829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.778549 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" event={"ID":"9e00115b-49bd-4a20-b83b-c845f2b06343","Type":"ContainerDied","Data":"0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b"} Nov 02 17:58:08.778829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.778639 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9" event={"ID":"9e00115b-49bd-4a20-b83b-c845f2b06343","Type":"ContainerDied","Data":"4e6fca03e584acf1c4d7548f0809280df24a59fea506a753794f1d40f81547e5"} Nov 02 17:58:08.778829 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.778670 5647 scope.go:122] "RemoveContainer" containerID="0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b" Nov 02 17:58:08.789986 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.789966 5647 scope.go:122] "RemoveContainer" containerID="4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f" Nov 02 17:58:08.799331 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.799313 5647 scope.go:122] "RemoveContainer" containerID="0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b" Nov 02 17:58:08.799630 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:08.799606 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b\": container with ID starting with 0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b not found: ID does not exist" containerID="0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b" Nov 02 17:58:08.799683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.799641 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b"} err="failed to get container status \"0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b\": rpc error: code = NotFound desc = could not find container \"0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b\": container with ID starting with 0eb4a48ae1039aefeeeca8bc40b446e6ca87c8142d3f170e4403a961571a763b not found: ID does not exist" Nov 02 17:58:08.799683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.799661 5647 scope.go:122] "RemoveContainer" containerID="4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f" Nov 02 17:58:08.799924 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:08.799906 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f\": container with ID starting with 4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f not found: ID does not exist" containerID="4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f" Nov 02 17:58:08.799971 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.799932 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f"} err="failed to get container status \"4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f\": rpc error: code = NotFound desc = could not find container \"4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f\": container with ID starting with 4e44c65e833af9fde4cbfe65ee834e30711819ba426522979bd689a1cdf1054f not found: ID does not exist" Nov 02 17:58:08.922761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.922119 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:58:08.922761 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.922197 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 02 17:58:08.959381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:08.959343 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9"] Nov 02 17:58:09.027410 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:09.027358 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-700abe-openshift-default-8ff44c697-hhtf9"] Nov 02 17:58:09.205047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:09.198854 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" path="/var/lib/kubelet/pods/9e00115b-49bd-4a20-b83b-c845f2b06343/volumes" Nov 02 17:58:09.919655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:09.919176 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:58:09.919655 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:09.919240 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 02 17:58:10.041405 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.041338 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:58:10.075468 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075439 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075669 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075494 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075669 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075518 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075669 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075637 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075775 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:10.075883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075785 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075898 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075934 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.075992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.075988 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.076132 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076017 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket\") pod \"10786ae4-769f-4e4e-a01a-e0354e2366bd\" (UID: \"10786ae4-769f-4e4e-a01a-e0354e2366bd\") " Nov 02 17:58:10.076187 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076146 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:10.076297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076262 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:10.076382 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076360 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:10.076607 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076399 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:10.076842 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076820 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.076921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076847 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.076921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076863 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.076921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076878 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10786ae4-769f-4e4e-a01a-e0354e2366bd-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.076921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.076893 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.078122 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.078096 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:10.078228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.078150 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "kube-api-access-nch8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:10.078228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.078166 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:10.078319 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.078229 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo" pod "10786ae4-769f-4e4e-a01a-e0354e2366bd" (UID: "10786ae4-769f-4e4e-a01a-e0354e2366bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:10.177855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.177821 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.177855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.177848 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nch8f\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-kube-api-access-nch8f\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.177855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.177858 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.178096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.177867 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10786ae4-769f-4e4e-a01a-e0354e2366bd-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:10.795019 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.794982 5647 generic.go:356] "Generic (PLEG): container finished" podID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerID="7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a" exitCode=0 Nov 02 17:58:10.795215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.795034 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" Nov 02 17:58:10.795215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.795062 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" event={"ID":"10786ae4-769f-4e4e-a01a-e0354e2366bd","Type":"ContainerDied","Data":"7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a"} Nov 02 17:58:10.795215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.795122 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf" event={"ID":"10786ae4-769f-4e4e-a01a-e0354e2366bd","Type":"ContainerDied","Data":"51e2d9f1354b38e5d02652edd02a179a809c8b3e4b6acc57daee67c5c7b1639b"} Nov 02 17:58:10.795215 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.795148 5647 scope.go:122] "RemoveContainer" containerID="7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a" Nov 02 17:58:10.805914 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.805892 5647 scope.go:122] "RemoveContainer" containerID="aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9" Nov 02 17:58:10.816210 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.816182 5647 scope.go:122] "RemoveContainer" containerID="7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a" Nov 02 17:58:10.816518 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:10.816501 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a\": container with ID starting with 7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a not found: ID does not exist" containerID="7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a" Nov 02 17:58:10.816572 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.816527 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a"} err="failed to get container status \"7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a\": rpc error: code = NotFound desc = could not find container \"7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a\": container with ID starting with 7db37911f6bb75623e953fb94cd1d9db45e552f16fbace625dad2f85f5ff331a not found: ID does not exist" Nov 02 17:58:10.816572 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.816545 5647 scope.go:122] "RemoveContainer" containerID="aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9" Nov 02 17:58:10.816870 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:10.816847 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9\": container with ID starting with aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9 not found: ID does not exist" containerID="aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9" Nov 02 17:58:10.816947 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.816874 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9"} err="failed to get container status \"aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9\": rpc error: code = NotFound desc = could not find container \"aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9\": container with ID starting with aea6e0a04498f135ba3398cf2643b1ffb0059f714d5b4de67218b9c78898acd9 not found: ID does not exist" Nov 02 17:58:10.913288 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.913247 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Nov 02 17:58:10.913481 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.913302 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Nov 02 17:58:10.972656 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:10.972614 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf"] Nov 02 17:58:11.016378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.016345 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5bf666-openshift-default-785fc74f5f-7h6qf"] Nov 02 17:58:11.089544 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.089506 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" path="/var/lib/kubelet/pods/10786ae4-769f-4e4e-a01a-e0354e2366bd/volumes" Nov 02 17:58:11.918883 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.918853 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:58:11.919294 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.919267 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:58:11.920210 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.920186 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 17:58:11.964326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.964293 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb"] Nov 02 17:58:11.964535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:11.964512 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" containerID="cri-o://7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807" gracePeriod=30 Nov 02 17:58:17.221078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.221051 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:58:17.277937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.277854 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.277937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.277901 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278016 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278053 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278141 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278182 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278213 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278213 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:17.278332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278305 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:17.278585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278327 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278378 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs\") pod \"af0eba83-6e44-47a3-8315-4b8eb360ed15\" (UID: \"af0eba83-6e44-47a3-8315-4b8eb360ed15\") " Nov 02 17:58:17.278585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278399 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:17.278778 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278605 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:17.278983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.278902 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:17.279341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.279151 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.279341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.279180 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.279341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.279195 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af0eba83-6e44-47a3-8315-4b8eb360ed15-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.279341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.279210 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.279341 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.279225 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.281310 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.281236 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:17.288554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.288228 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "kube-api-access-nvdpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:17.288554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.288213 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:17.288554 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.288498 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo" pod "af0eba83-6e44-47a3-8315-4b8eb360ed15" (UID: "af0eba83-6e44-47a3-8315-4b8eb360ed15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:17.380292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.380256 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.380292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.380285 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.380292 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.380296 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvdpn\" (UniqueName: \"kubernetes.io/projected/af0eba83-6e44-47a3-8315-4b8eb360ed15-kube-api-access-nvdpn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.380553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.380306 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af0eba83-6e44-47a3-8315-4b8eb360ed15-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:17.841837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.841799 5647 generic.go:356] "Generic (PLEG): container finished" podID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerID="7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807" exitCode=0 Nov 02 17:58:17.842045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.841864 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" Nov 02 17:58:17.842045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.841889 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" event={"ID":"af0eba83-6e44-47a3-8315-4b8eb360ed15","Type":"ContainerDied","Data":"7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807"} Nov 02 17:58:17.842045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.841940 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb" event={"ID":"af0eba83-6e44-47a3-8315-4b8eb360ed15","Type":"ContainerDied","Data":"f2ef81d9a6805d85679a8b47676d31ea946cfd97a4882489986fc5a89e34da41"} Nov 02 17:58:17.842045 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.841961 5647 scope.go:122] "RemoveContainer" containerID="7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807" Nov 02 17:58:17.852994 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.852966 5647 scope.go:122] "RemoveContainer" containerID="7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807" Nov 02 17:58:17.853297 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:17.853275 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807\": container with ID starting with 7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807 not found: ID does not exist" containerID="7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807" Nov 02 17:58:17.853366 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:17.853306 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807"} err="failed to get container status \"7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807\": rpc error: code = NotFound desc = could not find container \"7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807\": container with ID starting with 7e6e351d1978ca19fccf264b93b3706268906865bd58aa565b9f01b1d81b1807 not found: ID does not exist" Nov 02 17:58:18.010309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:18.010264 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb"] Nov 02 17:58:18.061016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:18.060981 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a099c3-openshift-default-668899b7f5-kmqtb"] Nov 02 17:58:19.081879 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.081847 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" path="/var/lib/kubelet/pods/af0eba83-6e44-47a3-8315-4b8eb360ed15/volumes" Nov 02 17:58:19.803535 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.803497 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r"] Nov 02 17:58:19.804100 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804084 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9491ac0a-2ae5-48cc-85a1-5856520fb4d5" containerName="payload-processing" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804102 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9491ac0a-2ae5-48cc-85a1-5856520fb4d5" containerName="payload-processing" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804124 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804129 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804144 5647 cpu_manager.go:414] "containerMap: removing container" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804149 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804157 5647 cpu_manager.go:414] "containerMap: removing container" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" Nov 02 17:58:19.804160 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804162 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804172 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1f35f43b-a6e2-4870-b563-bbbd8202df8d" containerName="payload-pre-processing" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804177 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f35f43b-a6e2-4870-b563-bbbd8202df8d" containerName="payload-pre-processing" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804198 5647 cpu_manager.go:414] "containerMap: removing container" podUID="94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" containerName="maas-api" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804203 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" containerName="maas-api" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804278 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9491ac0a-2ae5-48cc-85a1-5856520fb4d5" containerName="payload-processing" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804287 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804297 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804303 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0eba83-6e44-47a3-8315-4b8eb360ed15" containerName="istio-proxy" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804310 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804315 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804321 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f35f43b-a6e2-4870-b563-bbbd8202df8d" containerName="payload-pre-processing" Nov 02 17:58:19.804427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.804329 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fa3f47-4c8c-4fb6-a903-85502bb1ff4c" containerName="maas-api" Nov 02 17:58:19.807106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.807089 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:19.862297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.862259 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f2289f\"" Nov 02 17:58:19.951099 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.951063 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f2289f-dockercfg-82dm8\"" Nov 02 17:58:19.952886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.952204 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:19.953223 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.953203 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntxjh\" (UniqueName: \"kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh\") pod \"payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:19.953350 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.953322 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r"] Nov 02 17:58:19.983344 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.983306 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf"] Nov 02 17:58:19.984337 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.984308 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" Nov 02 17:58:19.984558 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.984520 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e00115b-49bd-4a20-b83b-c845f2b06343" containerName="istio-proxy" Nov 02 17:58:19.984706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.984568 5647 cpu_manager.go:414] "containerMap: removing container" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" Nov 02 17:58:19.984706 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.984576 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10786ae4-769f-4e4e-a01a-e0354e2366bd" containerName="istio-proxy" Nov 02 17:58:19.987234 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:19.987210 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.045853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.045812 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf"] Nov 02 17:58:20.054680 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.054596 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:20.054855 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.054771 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntxjh\" (UniqueName: \"kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh\") pod \"payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:20.055326 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.055302 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:20.113241 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.113204 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntxjh\" (UniqueName: \"kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh\") pod \"payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:20.120023 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.119995 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:20.156147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.156115 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.156457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.156437 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbgnz\" (UniqueName: \"kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz\") pod \"payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.257444 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.257405 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbgnz\" (UniqueName: \"kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz\") pod \"payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.257633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.257527 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.258266 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.258243 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.318117 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.318035 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbgnz\" (UniqueName: \"kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz\") pod \"payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.556549 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.556520 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r"] Nov 02 17:58:20.558273 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:58:20.558243 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa1b65b_9b54_41cc_b17f_0d0b9e823f40.slice/crio-78fb5565181b2f951b9549431e42ce7c8860bd6d731455531df828a37c73652c WatchSource:0}: Error finding container 78fb5565181b2f951b9549431e42ce7c8860bd6d731455531df828a37c73652c: Status 404 returned error can't find the container with id 78fb5565181b2f951b9549431e42ce7c8860bd6d731455531df828a37c73652c Nov 02 17:58:20.604565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.604535 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:20.864653 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.864616 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" event={"ID":"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40","Type":"ContainerStarted","Data":"b318e2591c979d72476f7c82e05004769c25625a2d3831e4499298c4b6bf5fa4"} Nov 02 17:58:20.864872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.864684 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" event={"ID":"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40","Type":"ContainerStarted","Data":"78fb5565181b2f951b9549431e42ce7c8860bd6d731455531df828a37c73652c"} Nov 02 17:58:20.864872 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.864848 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:20.982875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:20.982803 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" podStartSLOduration=1.982785959 podStartE2EDuration="1.982785959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:20.98154433 +0000 UTC m=+2088.055071535" watchObservedRunningTime="2026-11-02 17:58:20.982785959 +0000 UTC m=+2088.056313167" Nov 02 17:58:21.030368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:21.030336 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf"] Nov 02 17:58:21.032157 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:58:21.032125 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda114dd0_7138_4f17_b68c_e8c95e4a336e.slice/crio-fed4eeda8b08fd6e08355f2e2efadd762406a0bb4c7d89faa9da18c837504e99 WatchSource:0}: Error finding container fed4eeda8b08fd6e08355f2e2efadd762406a0bb4c7d89faa9da18c837504e99: Status 404 returned error can't find the container with id fed4eeda8b08fd6e08355f2e2efadd762406a0bb4c7d89faa9da18c837504e99 Nov 02 17:58:21.859004 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:21.858970 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx"] Nov 02 17:58:21.859465 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:21.859161 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" podUID="594e29d2-f6c2-4048-b33b-81f8a6c4b92c" containerName="maas-api" containerID="cri-o://a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334" gracePeriod=30 Nov 02 17:58:21.873594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:21.873564 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" event={"ID":"da114dd0-7138-4f17-b68c-e8c95e4a336e","Type":"ContainerStarted","Data":"06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356"} Nov 02 17:58:21.873798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:21.873615 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" event={"ID":"da114dd0-7138-4f17-b68c-e8c95e4a336e","Type":"ContainerStarted","Data":"fed4eeda8b08fd6e08355f2e2efadd762406a0bb4c7d89faa9da18c837504e99"} Nov 02 17:58:21.873798 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:21.873675 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:22.160065 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.160031 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:58:22.283585 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.283529 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" podStartSLOduration=3.283507357 podStartE2EDuration="3.283507357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:22.127161158 +0000 UTC m=+2089.200688362" watchObservedRunningTime="2026-11-02 17:58:22.283507357 +0000 UTC m=+2089.357034561" Nov 02 17:58:22.286415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.284647 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc\" (UniqueName: \"kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc\") pod \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " Nov 02 17:58:22.286415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.284690 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca\") pod \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " Nov 02 17:58:22.286415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.284845 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs\") pod \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " Nov 02 17:58:22.286415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.284928 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\") pod \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\" (UID: \"594e29d2-f6c2-4048-b33b-81f8a6c4b92c\") " Nov 02 17:58:22.286415 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.286126 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca" pod "594e29d2-f6c2-4048-b33b-81f8a6c4b92c" (UID: "594e29d2-f6c2-4048-b33b-81f8a6c4b92c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:22.290520 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.290489 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs" pod "594e29d2-f6c2-4048-b33b-81f8a6c4b92c" (UID: "594e29d2-f6c2-4048-b33b-81f8a6c4b92c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:58:22.290892 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.290852 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls" pod "594e29d2-f6c2-4048-b33b-81f8a6c4b92c" (UID: "594e29d2-f6c2-4048-b33b-81f8a6c4b92c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:58:22.291173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.291143 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc" pod "594e29d2-f6c2-4048-b33b-81f8a6c4b92c" (UID: "594e29d2-f6c2-4048-b33b-81f8a6c4b92c"). InnerVolumeSpecName "kube-api-access-2j7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:22.387289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.387204 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7fc\" (UniqueName: \"kubernetes.io/projected/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-kube-api-access-2j7fc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:22.387289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.387235 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:22.387289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.387248 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:22.387289 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.387256 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/594e29d2-f6c2-4048-b33b-81f8a6c4b92c-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:22.882329 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.882297 5647 generic.go:356] "Generic (PLEG): container finished" podID="594e29d2-f6c2-4048-b33b-81f8a6c4b92c" containerID="a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334" exitCode=0 Nov 02 17:58:22.882821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.882339 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" Nov 02 17:58:22.882821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.882374 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" event={"ID":"594e29d2-f6c2-4048-b33b-81f8a6c4b92c","Type":"ContainerDied","Data":"a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334"} Nov 02 17:58:22.882821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.882425 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx" event={"ID":"594e29d2-f6c2-4048-b33b-81f8a6c4b92c","Type":"ContainerDied","Data":"bbfb5fcc48e61338511fea6ad20b9595e0db7329c536acbc6844472a2d528870"} Nov 02 17:58:22.882821 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.882456 5647 scope.go:122] "RemoveContainer" containerID="a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334" Nov 02 17:58:22.894489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.894465 5647 scope.go:122] "RemoveContainer" containerID="a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334" Nov 02 17:58:22.894798 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:22.894776 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334\": container with ID starting with a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334 not found: ID does not exist" containerID="a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334" Nov 02 17:58:22.894853 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:22.894807 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334"} err="failed to get container status \"a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334\": rpc error: code = NotFound desc = could not find container \"a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334\": container with ID starting with a55dafd7beea925d1c6b7e45b5b5171a1683b2ef4406572111a7e373b831a334 not found: ID does not exist" Nov 02 17:58:23.056076 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:23.056040 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx"] Nov 02 17:58:23.060888 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:23.060863 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-9ac699-7546b99cd7-7t7cx"] Nov 02 17:58:23.083296 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:23.083263 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594e29d2-f6c2-4048-b33b-81f8a6c4b92c" path="/var/lib/kubelet/pods/594e29d2-f6c2-4048-b33b-81f8a6c4b92c/volumes" Nov 02 17:58:25.054674 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:25.054641 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:58:29.287543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:29.287503 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn"] Nov 02 17:58:29.288133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:29.287859 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" containerID="cri-o://d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f" gracePeriod=30 Nov 02 17:58:31.430519 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.430474 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g"] Nov 02 17:58:31.431378 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.431353 5647 cpu_manager.go:414] "containerMap: removing container" podUID="594e29d2-f6c2-4048-b33b-81f8a6c4b92c" containerName="maas-api" Nov 02 17:58:31.431485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.431380 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594e29d2-f6c2-4048-b33b-81f8a6c4b92c" containerName="maas-api" Nov 02 17:58:31.431553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.431526 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="594e29d2-f6c2-4048-b33b-81f8a6c4b92c" containerName="maas-api" Nov 02 17:58:31.435235 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.435212 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.492179 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.492142 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-fd04fa-openshift-default-dockercfg-vhjll\"" Nov 02 17:58:31.564131 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564086 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564141 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564172 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxz2g\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564287 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564343 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564336 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564413 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564476 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564513 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.564570 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.564550 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.565846 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.565820 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g"] Nov 02 17:58:31.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669543 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669649 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669681 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669708 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxz2g\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669850 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669894 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.670427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.669960 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.670427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.670014 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.670427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.670047 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.670591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.670501 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.670791 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.670769 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.671043 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.671020 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.671114 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.671086 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.671673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.671643 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.673880 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.673798 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.679309 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.679277 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.738413 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.738277 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.740186 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.740089 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxz2g\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g\") pod \"e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.772532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.772501 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:31.878969 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:31.877874 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 17:58:32.212075 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.212042 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g"] Nov 02 17:58:32.214257 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:58:32.214229 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531cd99d_2559_464f_899d_88a9f483efe2.slice/crio-af7766fbca067116953a177fc66f2e1a3c82819183c872cd873ef9930fb1c754 WatchSource:0}: Error finding container af7766fbca067116953a177fc66f2e1a3c82819183c872cd873ef9930fb1c754: Status 404 returned error can't find the container with id af7766fbca067116953a177fc66f2e1a3c82819183c872cd873ef9930fb1c754 Nov 02 17:58:32.217130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.217088 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:32.217250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.217180 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:32.217250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.217231 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:32.886061 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.886026 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 17:58:32.986463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.986417 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" event={"ID":"531cd99d-2559-464f-899d-88a9f483efe2","Type":"ContainerStarted","Data":"ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c"} Nov 02 17:58:32.986678 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:32.986485 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" event={"ID":"531cd99d-2559-464f-899d-88a9f483efe2","Type":"ContainerStarted","Data":"af7766fbca067116953a177fc66f2e1a3c82819183c872cd873ef9930fb1c754"} Nov 02 17:58:33.143940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:33.143820 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podStartSLOduration=2.143799439 podStartE2EDuration="2.143799439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:33.141315578 +0000 UTC m=+2100.214842781" watchObservedRunningTime="2026-11-02 17:58:33.143799439 +0000 UTC m=+2100.217326643" Nov 02 17:58:33.772801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:33.772757 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:33.779945 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:33.779911 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:33.994954 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:33.994921 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:33.995907 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:33.995884 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 17:58:34.197934 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.197906 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:58:34.203998 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.203961 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 17:58:34.569262 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.569236 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:58:34.611149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611117 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611149 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611157 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611190 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611361 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611417 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611408 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611476 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:34.611593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611557 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611593 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611569 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:58:34.611709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611614 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611659 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.611912 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611863 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:34.611993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611918 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:34.611993 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.611953 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:34.612538 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.612511 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.612649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.612541 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.612649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.612556 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.612649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.612570 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff3cf331-f2be-42c5-8866-f9cf477086f5-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.612649 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.612584 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.613926 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.613894 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:34.614024 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.613960 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "kube-api-access-n5x49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:58:34.614147 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.614131 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:58:34.713526 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.713489 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy\") pod \"ff3cf331-f2be-42c5-8866-f9cf477086f5\" (UID: \"ff3cf331-f2be-42c5-8866-f9cf477086f5\") " Nov 02 17:58:34.713902 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.713886 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.713968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.713903 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5x49\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-kube-api-access-n5x49\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.713968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.713914 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:34.715836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.715805 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy" pod "ff3cf331-f2be-42c5-8866-f9cf477086f5" (UID: "ff3cf331-f2be-42c5-8866-f9cf477086f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:58:34.815482 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:34.815403 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff3cf331-f2be-42c5-8866-f9cf477086f5-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:58:35.004284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.004250 5647 generic.go:356] "Generic (PLEG): container finished" podID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerID="d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f" exitCode=0 Nov 02 17:58:35.004698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.004334 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" Nov 02 17:58:35.004698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.004340 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" event={"ID":"ff3cf331-f2be-42c5-8866-f9cf477086f5","Type":"ContainerDied","Data":"d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f"} Nov 02 17:58:35.004698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.004420 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn" event={"ID":"ff3cf331-f2be-42c5-8866-f9cf477086f5","Type":"ContainerDied","Data":"46891db4022459fa03f26766a90e24583b0f3e73551bf21e061d10fcae783255"} Nov 02 17:58:35.004698 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.004458 5647 scope.go:122] "RemoveContainer" containerID="d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f" Nov 02 17:58:35.015602 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.015574 5647 scope.go:122] "RemoveContainer" containerID="2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e" Nov 02 17:58:35.027979 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.027955 5647 scope.go:122] "RemoveContainer" containerID="d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f" Nov 02 17:58:35.028380 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:35.028352 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f\": container with ID starting with d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f not found: ID does not exist" containerID="d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f" Nov 02 17:58:35.028447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.028397 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f"} err="failed to get container status \"d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f\": rpc error: code = NotFound desc = could not find container \"d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f\": container with ID starting with d3d9c0df8e35967fa1aa7d2e0d32d5b44b22516dd2b01718ab1b44b1ff4a6f9f not found: ID does not exist" Nov 02 17:58:35.028447 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.028428 5647 scope.go:122] "RemoveContainer" containerID="2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e" Nov 02 17:58:35.028746 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:35.028710 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e\": container with ID starting with 2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e not found: ID does not exist" containerID="2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e" Nov 02 17:58:35.028838 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.028778 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e"} err="failed to get container status \"2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e\": rpc error: code = NotFound desc = could not find container \"2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e\": container with ID starting with 2a6ba260d3331cadbec859f0c3dd77c21ab73ccdc5e6d5ee9762b9bda5965b9e not found: ID does not exist" Nov 02 17:58:35.176416 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.176371 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn"] Nov 02 17:58:35.225436 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:35.225404 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-9ac699-openshift-default-65455bd5c9-v2gsn"] Nov 02 17:58:35.948587 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:58:35.948552 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:58:36.012035 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:36.012001 5647 generic.go:356] "Generic (PLEG): container finished" podID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerID="31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f" exitCode=0 Nov 02 17:58:36.012497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:36.012082 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" event={"ID":"653639db-490a-41c7-9ef1-a067d45c5f7b","Type":"ContainerDied","Data":"31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f"} Nov 02 17:58:36.012620 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:36.012601 5647 scope.go:122] "RemoveContainer" containerID="31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f" Nov 02 17:58:36.014155 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:36.014127 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:36.014229 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:36.014196 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:36.014270 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:36.014229 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:37.034245 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:37.034208 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" event={"ID":"653639db-490a-41c7-9ef1-a067d45c5f7b","Type":"ContainerStarted","Data":"d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b"} Nov 02 17:58:37.081882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:37.081849 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" path="/var/lib/kubelet/pods/ff3cf331-f2be-42c5-8866-f9cf477086f5/volumes" Nov 02 17:58:37.864555 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:37.864514 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:58:37.865940 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:37.865918 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 17:58:37.866011 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:37.865963 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 17:58:38.864673 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:38.864635 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 17:58:38.865173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:38.864689 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 17:58:39.648284 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.648247 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6"] Nov 02 17:58:39.648878 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.648862 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" Nov 02 17:58:39.648929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.648880 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" Nov 02 17:58:39.648929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.648893 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" Nov 02 17:58:39.648929 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.648899 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" Nov 02 17:58:39.649042 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.649031 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" Nov 02 17:58:39.649078 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.649044 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff3cf331-f2be-42c5-8866-f9cf477086f5" containerName="istio-proxy" Nov 02 17:58:39.653895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.653878 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.687005 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.686965 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.687139 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.687111 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7gs6\" (UniqueName: \"kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6\") pod \"payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.731759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.727803 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-fd04fa\"" Nov 02 17:58:39.731973 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.731942 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6"] Nov 02 17:58:39.791426 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.789963 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.800802 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.800767 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7gs6\" (UniqueName: \"kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6\") pod \"payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.801886 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.801131 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.804817 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.804789 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc"] Nov 02 17:58:39.869000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.857061 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:39.869000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.863470 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc"] Nov 02 17:58:39.869000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.868435 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7gs6\" (UniqueName: \"kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6\") pod \"payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:39.869000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.868829 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 17:58:39.869000 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.868885 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 17:58:39.927813 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.905605 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6c26\" (UniqueName: \"kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26\") pod \"payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:39.928360 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.928327 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:39.928627 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.922151 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-fd04fa-dockercfg-bsl5z\"" Nov 02 17:58:39.967298 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:39.967264 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:40.029759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.028872 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:58:40.029759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.029550 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6c26\" (UniqueName: \"kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26\") pod \"payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:40.029759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.029681 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:40.030928 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.030393 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:40.090613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.090581 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6c26\" (UniqueName: \"kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26\") pod \"payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:40.198106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.198014 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:40.468875 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.453963 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6"] Nov 02 17:58:40.629079 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.622526 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc"] Nov 02 17:58:40.864612 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.864579 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 17:58:40.864796 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:40.864632 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 17:58:41.061553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.061451 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" event={"ID":"6e43c32f-b34b-4059-841a-f0f7eb578eb5","Type":"ContainerStarted","Data":"1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862"} Nov 02 17:58:41.061553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.061514 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" event={"ID":"6e43c32f-b34b-4059-841a-f0f7eb578eb5","Type":"ContainerStarted","Data":"af30cedbe41bc6799676df28c3354037557cdf21cd7b2722273e4ddaf616bb7c"} Nov 02 17:58:41.061553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.061542 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:41.067381 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.067340 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" event={"ID":"a84587b3-79b0-441d-b159-59411f0d343f","Type":"ContainerStarted","Data":"9c1e13c955ebb9cc1d067dc6d965abca3c4990af26306c119d18c06b98db6ff0"} Nov 02 17:58:41.067562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.067405 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" event={"ID":"a84587b3-79b0-441d-b159-59411f0d343f","Type":"ContainerStarted","Data":"7ab9c8b0c55a3e34f917dee4ac5c5d84c90ff7551fb2cad5e7122ad47339a391"} Nov 02 17:58:41.067562 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.067437 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:58:41.194063 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.193997 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" podStartSLOduration=2.193981103 podStartE2EDuration="2.193981103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:41.192268993 +0000 UTC m=+2108.265796196" watchObservedRunningTime="2026-11-02 17:58:41.193981103 +0000 UTC m=+2108.267508305" Nov 02 17:58:41.864532 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.864489 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Nov 02 17:58:41.864726 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:41.864555 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Nov 02 17:58:42.868466 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:42.868437 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:58:42.869548 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:42.869530 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 17:58:43.047143 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:43.047085 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" podStartSLOduration=4.047069392 podStartE2EDuration="4.047069392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:41.329121728 +0000 UTC m=+2108.402648932" watchObservedRunningTime="2026-11-02 17:58:43.047069392 +0000 UTC m=+2110.120596595" Nov 02 17:58:48.254193 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.254156 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt"] Nov 02 17:58:48.258489 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.258471 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.312956 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.312910 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f42381-openshift-default-dockercfg-442wq\"" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350533 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350585 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350639 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350671 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350712 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350778 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czv5h\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350832 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350903 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.353686 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.350949 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.377845 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.367424 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt"] Nov 02 17:58:48.451962 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.451922 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.451977 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452057 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452077 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452111 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452151 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452130 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452160 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452193 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czv5h\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452230 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452457 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452338 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452506 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452688 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452654 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.452871 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.452849 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.453173 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.453152 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.454783 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.454761 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.455297 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.455278 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.511428 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.511229 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.513087 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.513062 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czv5h\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h\") pod \"e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:48.573272 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:48.573239 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:49.036438 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.036408 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt"] Nov 02 17:58:49.038132 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:58:49.038102 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fd7905_7063_46bf_9e1f_1d7755094293.slice/crio-ffc949ffd2db694a19498340a4fe45d75d20cd58259c41056b84c72e3941b239 WatchSource:0}: Error finding container ffc949ffd2db694a19498340a4fe45d75d20cd58259c41056b84c72e3941b239: Status 404 returned error can't find the container with id ffc949ffd2db694a19498340a4fe45d75d20cd58259c41056b84c72e3941b239 Nov 02 17:58:49.040790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.040755 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:49.040895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.040825 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:49.040895 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.040875 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:58:49.149279 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.149241 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" event={"ID":"89fd7905-7063-46bf-9e1f-1d7755094293","Type":"ContainerStarted","Data":"938e9e2a92eb3e4029809a35c41949c4c9a9db74c63bdcc6cb30c9a1aa7d6092"} Nov 02 17:58:49.149440 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.149298 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" event={"ID":"89fd7905-7063-46bf-9e1f-1d7755094293","Type":"ContainerStarted","Data":"ffc949ffd2db694a19498340a4fe45d75d20cd58259c41056b84c72e3941b239"} Nov 02 17:58:49.271401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.271332 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podStartSLOduration=1.271318557 podStartE2EDuration="1.271318557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:49.270073197 +0000 UTC m=+2116.343600423" watchObservedRunningTime="2026-11-02 17:58:49.271318557 +0000 UTC m=+2116.344845760" Nov 02 17:58:49.573915 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:49.573881 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:50.581106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:50.581074 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:51.164123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:51.164089 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:51.165103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:51.165083 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 17:58:52.076958 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:52.076924 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 17:58:52.077403 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:58:52.077000 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 17:59:01.184867 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:01.184828 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk"] Nov 02 17:59:01.289141 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:01.289109 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89"] Nov 02 17:59:05.948606 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:05.948568 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:59:06.185921 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.185859 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" podUID="f3093163-fb0a-460b-881a-f716151ae3f8" containerName="payload-processing" containerID="cri-o://7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5" gracePeriod=25 Nov 02 17:59:06.289651 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.289598 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" podUID="848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" containerName="payload-pre-processing" containerID="cri-o://253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1" gracePeriod=25 Nov 02 17:59:06.503320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.503290 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:59:06.626332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.626299 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:59:06.650644 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.650612 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q\" (UniqueName: \"kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q\") pod \"f3093163-fb0a-460b-881a-f716151ae3f8\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " Nov 02 17:59:06.650869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.650712 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume\") pod \"f3093163-fb0a-460b-881a-f716151ae3f8\" (UID: \"f3093163-fb0a-460b-881a-f716151ae3f8\") " Nov 02 17:59:06.651106 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.651079 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume" pod "f3093163-fb0a-460b-881a-f716151ae3f8" (UID: "f3093163-fb0a-460b-881a-f716151ae3f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:06.653040 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.653013 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q" pod "f3093163-fb0a-460b-881a-f716151ae3f8" (UID: "f3093163-fb0a-460b-881a-f716151ae3f8"). InnerVolumeSpecName "kube-api-access-dsm6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:06.675406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.675369 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f"] Nov 02 17:59:06.675594 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.675574 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" podUID="b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" containerName="maas-api" containerID="cri-o://b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83" gracePeriod=30 Nov 02 17:59:06.752401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.752300 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d\" (UniqueName: \"kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d\") pod \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " Nov 02 17:59:06.752630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.752498 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume\") pod \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\" (UID: \"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8\") " Nov 02 17:59:06.753058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.753029 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsm6q\" (UniqueName: \"kubernetes.io/projected/f3093163-fb0a-460b-881a-f716151ae3f8-kube-api-access-dsm6q\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:06.753058 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.753060 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3093163-fb0a-460b-881a-f716151ae3f8-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:06.753498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.753472 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume" pod "848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" (UID: "848412c4-4841-4fd2-ac35-a8d1c4a3b5c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:06.756551 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.756524 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d" pod "848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" (UID: "848412c4-4841-4fd2-ac35-a8d1c4a3b5c8"). InnerVolumeSpecName "kube-api-access-fc56d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:06.785624 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.785589 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:59:06.854363 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.854336 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:06.854464 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.854367 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc56d\" (UniqueName: \"kubernetes.io/projected/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8-kube-api-access-fc56d\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:06.941332 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:06.941302 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:59:07.057165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.057078 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\") pod \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " Nov 02 17:59:07.057165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.057114 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs\") pod \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " Nov 02 17:59:07.057165 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.057143 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq\" (UniqueName: \"kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq\") pod \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " Nov 02 17:59:07.058017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.057282 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca\") pod \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\" (UID: \"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60\") " Nov 02 17:59:07.058017 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.057652 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca" pod "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" (UID: "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:07.059152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.059129 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:07.060449 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.059892 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls" pod "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" (UID: "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:07.060683 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.060525 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs" pod "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" (UID: "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 17:59:07.061166 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.061139 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq" pod "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" (UID: "b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60"). InnerVolumeSpecName "kube-api-access-qddkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:07.160406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.160374 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:07.160406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.160398 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:07.160406 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.160408 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qddkq\" (UniqueName: \"kubernetes.io/projected/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60-kube-api-access-qddkq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:07.264968 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.264932 5647 generic.go:356] "Generic (PLEG): container finished" podID="b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" containerID="b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83" exitCode=0 Nov 02 17:59:07.265161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.264987 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" Nov 02 17:59:07.265161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.265030 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" event={"ID":"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60","Type":"ContainerDied","Data":"b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83"} Nov 02 17:59:07.265161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.265090 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f" event={"ID":"b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60","Type":"ContainerDied","Data":"7a0312af9d51a65322780e7771c0224561d522a51d88a7109d1ceda57fe2311c"} Nov 02 17:59:07.265161 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.265118 5647 scope.go:122] "RemoveContainer" containerID="b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83" Nov 02 17:59:07.266613 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.266591 5647 generic.go:356] "Generic (PLEG): container finished" podID="f3093163-fb0a-460b-881a-f716151ae3f8" containerID="7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5" exitCode=0 Nov 02 17:59:07.266705 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.266664 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" Nov 02 17:59:07.266705 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.266683 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" event={"ID":"f3093163-fb0a-460b-881a-f716151ae3f8","Type":"ContainerDied","Data":"7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5"} Nov 02 17:59:07.266911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.266726 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk" event={"ID":"f3093163-fb0a-460b-881a-f716151ae3f8","Type":"ContainerDied","Data":"1051ce4ab1cd6714de3a0efb0f8fb41d7f1c8c9fd00d7b755e2f1b7e5ae9f18c"} Nov 02 17:59:07.268974 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.268948 5647 generic.go:356] "Generic (PLEG): container finished" podID="848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" containerID="253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1" exitCode=0 Nov 02 17:59:07.269116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.268997 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" Nov 02 17:59:07.269116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.269022 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" event={"ID":"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8","Type":"ContainerDied","Data":"253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1"} Nov 02 17:59:07.269116 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.269063 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89" event={"ID":"848412c4-4841-4fd2-ac35-a8d1c4a3b5c8","Type":"ContainerDied","Data":"52b2ec70de14701d8dc70b98352036039c7b2b67e79dbd7e07c395edcab41ef4"} Nov 02 17:59:07.276677 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.276661 5647 scope.go:122] "RemoveContainer" containerID="b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83" Nov 02 17:59:07.276977 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:07.276957 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83\": container with ID starting with b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83 not found: ID does not exist" containerID="b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83" Nov 02 17:59:07.277082 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.276985 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83"} err="failed to get container status \"b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83\": rpc error: code = NotFound desc = could not find container \"b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83\": container with ID starting with b29410fc6fe6cae94dc01786d91d9107ebdd0007803825a2037272810918df83 not found: ID does not exist" Nov 02 17:59:07.277082 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.277003 5647 scope.go:122] "RemoveContainer" containerID="7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5" Nov 02 17:59:07.293997 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.293957 5647 scope.go:122] "RemoveContainer" containerID="7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5" Nov 02 17:59:07.294578 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:07.294537 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5\": container with ID starting with 7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5 not found: ID does not exist" containerID="7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5" Nov 02 17:59:07.294715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.294580 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5"} err="failed to get container status \"7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5\": rpc error: code = NotFound desc = could not find container \"7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5\": container with ID starting with 7dd158509ca5b731d1e94128c5fb4a38a1606784b41a170c2e28f55e1659e5a5 not found: ID does not exist" Nov 02 17:59:07.294715 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.294610 5647 scope.go:122] "RemoveContainer" containerID="253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1" Nov 02 17:59:07.306861 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.306840 5647 scope.go:122] "RemoveContainer" containerID="253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1" Nov 02 17:59:07.307194 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:07.307145 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1\": container with ID starting with 253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1 not found: ID does not exist" containerID="253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1" Nov 02 17:59:07.307278 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.307180 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1"} err="failed to get container status \"253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1\": rpc error: code = NotFound desc = could not find container \"253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1\": container with ID starting with 253d52fae55338b492af16de250eaf20871f3fb2eacecacaf15a9a0ec490aed1 not found: ID does not exist" Nov 02 17:59:07.441970 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.441935 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89"] Nov 02 17:59:07.446837 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.446813 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d5924c-bbf7df779-bdc89"] Nov 02 17:59:07.556936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.556901 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f"] Nov 02 17:59:07.560595 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.560529 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cea7c-b95d4f879-gzv9f"] Nov 02 17:59:07.676882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.676847 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk"] Nov 02 17:59:07.683530 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:07.683502 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d5924c-66c459d9d9-2nnlk"] Nov 02 17:59:09.081067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:09.081031 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" path="/var/lib/kubelet/pods/848412c4-4841-4fd2-ac35-a8d1c4a3b5c8/volumes" Nov 02 17:59:09.081484 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:09.081469 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" path="/var/lib/kubelet/pods/b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60/volumes" Nov 02 17:59:09.081843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:09.081830 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3093163-fb0a-460b-881a-f716151ae3f8" path="/var/lib/kubelet/pods/f3093163-fb0a-460b-881a-f716151ae3f8/volumes" Nov 02 17:59:11.620498 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:11.620462 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6"] Nov 02 17:59:11.620992 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:11.620714 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" containerID="cri-o://3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d" gracePeriod=30 Nov 02 17:59:13.115308 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115273 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4"] Nov 02 17:59:13.115924 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115907 5647 cpu_manager.go:414] "containerMap: removing container" podUID="848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" containerName="payload-pre-processing" Nov 02 17:59:13.115983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115926 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" containerName="payload-pre-processing" Nov 02 17:59:13.115983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115946 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f3093163-fb0a-460b-881a-f716151ae3f8" containerName="payload-processing" Nov 02 17:59:13.115983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115952 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3093163-fb0a-460b-881a-f716151ae3f8" containerName="payload-processing" Nov 02 17:59:13.115983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115966 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" containerName="maas-api" Nov 02 17:59:13.115983 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.115982 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" containerName="maas-api" Nov 02 17:59:13.116152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.116072 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41315a8-7dfc-4dfd-8fb7-0ee1cf38ae60" containerName="maas-api" Nov 02 17:59:13.116152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.116091 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="848412c4-4841-4fd2-ac35-a8d1c4a3b5c8" containerName="payload-pre-processing" Nov 02 17:59:13.116152 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.116098 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3093163-fb0a-460b-881a-f716151ae3f8" containerName="payload-processing" Nov 02 17:59:13.121056 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.121037 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.209759 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.178104 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f07b5b-openshift-default-dockercfg-v9njm\"" Nov 02 17:59:13.234123 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.234075 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4"] Nov 02 17:59:13.254879 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.254838 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255103 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255027 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255285 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255264 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255490 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255526 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjvbw\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255562 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255614 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255583 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.255936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.255842 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.356625 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356580 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.356851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356661 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.356851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356742 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.356851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356776 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjvbw\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.356851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356813 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.356851 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356837 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356871 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.356948 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.357002 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357128 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.357090 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.357135 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357354 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.357294 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357462 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.357409 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.357710 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.357691 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.359463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.359444 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.359540 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.359524 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.417456 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.417383 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.417456 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.417444 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjvbw\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw\") pod \"e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.439013 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.438983 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:13.788858 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.788768 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7"] Nov 02 17:59:13.793975 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.793948 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.854238 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.854192 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f2289f-serving-cert\"" Nov 02 17:59:13.856588 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.856561 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7"] Nov 02 17:59:13.878358 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.878314 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.878536 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.878436 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnbgr\" (UniqueName: \"kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.878624 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.878606 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.878693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.878657 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.925547 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:59:13.925509 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52cd3067_1b2c_492c_a7d8_e28d39754a5a.slice/crio-56e71ce2e47717ebeabdb727cfe99e213f5ba39cb7df827095e1dd2e6ab4b7ba WatchSource:0}: Error finding container 56e71ce2e47717ebeabdb727cfe99e213f5ba39cb7df827095e1dd2e6ab4b7ba: Status 404 returned error can't find the container with id 56e71ce2e47717ebeabdb727cfe99e213f5ba39cb7df827095e1dd2e6ab4b7ba Nov 02 17:59:13.927047 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.927021 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4"] Nov 02 17:59:13.929086 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.929041 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:59:13.929194 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.929156 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:59:13.929268 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.929203 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 17:59:13.979771 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.979722 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.979891 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.979837 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnbgr\" (UniqueName: \"kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.980015 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.980000 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.980067 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.980050 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.980175 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:13.980154 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-serving-cert: secret "maas-api-e2e-praxis-b-f2289f-serving-cert" not found Nov 02 17:59:13.980250 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:13.980239 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls podName:03d5b318-a3d4-456d-8eda-3c7951b1b426 nodeName:}" failed. No retries permitted until 2026-11-02 17:59:14.480220898 +0000 UTC m=+2141.553748085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls") pod "maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" (UID: "03d5b318-a3d4-456d-8eda-3c7951b1b426") : secret "maas-api-e2e-praxis-b-f2289f-serving-cert" not found Nov 02 17:59:13.980411 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.980389 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:13.982511 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:13.982489 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:14.064850 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.061330 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnbgr\" (UniqueName: \"kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:14.326038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.325940 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" event={"ID":"52cd3067-1b2c-492c-a7d8-e28d39754a5a","Type":"ContainerStarted","Data":"f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9"} Nov 02 17:59:14.326038 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.326009 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" event={"ID":"52cd3067-1b2c-492c-a7d8-e28d39754a5a","Type":"ContainerStarted","Data":"56e71ce2e47717ebeabdb727cfe99e213f5ba39cb7df827095e1dd2e6ab4b7ba"} Nov 02 17:59:14.439228 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.439197 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:14.445943 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.445919 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:14.450028 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.449979 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podStartSLOduration=1.449964584 podStartE2EDuration="1.449964584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:14.448929456 +0000 UTC m=+2141.522456672" watchObservedRunningTime="2026-11-02 17:59:14.449964584 +0000 UTC m=+2141.523491786" Nov 02 17:59:14.485911 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.485870 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:14.488577 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.488554 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:14.708616 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:14.708583 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:15.088096 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:15.088071 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7"] Nov 02 17:59:15.090961 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:59:15.090933 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d5b318_a3d4_456d_8eda_3c7951b1b426.slice/crio-fed5971bc7abfac8ed0c0e4b09970a7403018777cbc8510085612e5429eb6493 WatchSource:0}: Error finding container fed5971bc7abfac8ed0c0e4b09970a7403018777cbc8510085612e5429eb6493: Status 404 returned error can't find the container with id fed5971bc7abfac8ed0c0e4b09970a7403018777cbc8510085612e5429eb6493 Nov 02 17:59:15.333146 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:15.333113 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" event={"ID":"03d5b318-a3d4-456d-8eda-3c7951b1b426","Type":"ContainerStarted","Data":"fed5971bc7abfac8ed0c0e4b09970a7403018777cbc8510085612e5429eb6493"} Nov 02 17:59:15.333557 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:15.333465 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:15.334497 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:15.334473 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 17:59:16.340767 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.340699 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" event={"ID":"03d5b318-a3d4-456d-8eda-3c7951b1b426","Type":"ContainerStarted","Data":"a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4"} Nov 02 17:59:16.341282 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.340967 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:16.459055 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.459003 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" podStartSLOduration=2.584582477 podStartE2EDuration="3.458987499s" totalImagesPullingTime="874.405022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:16.456165983 +0000 UTC m=+2143.529693186" watchObservedRunningTime="2026-11-02 17:59:16.458987499 +0000 UTC m=+2143.532514701" Nov 02 17:59:16.904385 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.904360 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:59:16.924572 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924535 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.924723 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924633 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.924723 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924672 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.924723 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924700 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.924919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924772 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.924919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924829 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:16.925009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924932 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:16.925009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924949 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.925009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924980 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.925009 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.924948 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:16.925197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925057 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.925197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925100 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo\") pod \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\" (UID: \"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20\") " Nov 02 17:59:16.925197 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925123 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:16.925396 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925374 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 17:59:16.925619 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925603 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:16.925695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925625 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:16.925695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925641 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:16.925695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925654 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:16.925695 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.925669 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:16.927133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.927104 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "kube-api-access-sjgk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:16.927257 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.927225 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 17:59:16.927539 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.927518 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 17:59:16.927600 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:16.927539 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy" pod "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" (UID: "8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 17:59:17.026823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.026788 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:17.026823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.026814 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:17.026823 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.026825 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:17.027135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.026841 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjgk6\" (UniqueName: \"kubernetes.io/projected/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20-kube-api-access-sjgk6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 17:59:17.349581 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.349542 5647 generic.go:356] "Generic (PLEG): container finished" podID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerID="3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d" exitCode=0 Nov 02 17:59:17.350133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.349603 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" Nov 02 17:59:17.350133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.349620 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" event={"ID":"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20","Type":"ContainerDied","Data":"3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d"} Nov 02 17:59:17.350133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.349677 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6" event={"ID":"8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20","Type":"ContainerDied","Data":"1d9ffa48d6f6f24905c961615c0db4f3d8a56da4feea7d2de68ce982ccdd4e12"} Nov 02 17:59:17.350133 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.349701 5647 scope.go:122] "RemoveContainer" containerID="3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d" Nov 02 17:59:17.361565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.361546 5647 scope.go:122] "RemoveContainer" containerID="612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1" Nov 02 17:59:17.371159 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.371132 5647 scope.go:122] "RemoveContainer" containerID="3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d" Nov 02 17:59:17.371494 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:17.371473 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d\": container with ID starting with 3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d not found: ID does not exist" containerID="3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d" Nov 02 17:59:17.371582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.371508 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d"} err="failed to get container status \"3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d\": rpc error: code = NotFound desc = could not find container \"3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d\": container with ID starting with 3b4a158648343422727c41f2700196e3c347603d507455cb40932ada2a0f7b0d not found: ID does not exist" Nov 02 17:59:17.371582 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.371534 5647 scope.go:122] "RemoveContainer" containerID="612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1" Nov 02 17:59:17.371861 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:17.371837 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1\": container with ID starting with 612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1 not found: ID does not exist" containerID="612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1" Nov 02 17:59:17.371963 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.371866 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1"} err="failed to get container status \"612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1\": rpc error: code = NotFound desc = could not find container \"612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1\": container with ID starting with 612216a22eb985f2a46686ad4b004f820aa53eb7f056b5b66963a19c0b04b3f1 not found: ID does not exist" Nov 02 17:59:17.529776 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.529720 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6"] Nov 02 17:59:17.579565 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:17.579520 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1cea7c-openshift-default-86b9c445b7-7wfj6"] Nov 02 17:59:19.081304 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:19.081269 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" path="/var/lib/kubelet/pods/8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20/volumes" Nov 02 17:59:22.356401 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:22.356369 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 17:59:23.017459 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.017427 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss"] Nov 02 17:59:23.018041 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.018028 5647 cpu_manager.go:414] "containerMap: removing container" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" Nov 02 17:59:23.018107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.018043 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" Nov 02 17:59:23.018107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.018060 5647 cpu_manager.go:414] "containerMap: removing container" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" Nov 02 17:59:23.018107 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.018065 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" Nov 02 17:59:23.018207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.018171 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" Nov 02 17:59:23.018207 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.018180 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7b82c1-e862-4aa5-9032-8c3a6ceb3c20" containerName="istio-proxy" Nov 02 17:59:23.023130 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.023113 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.101800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.093925 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f07b5b-dockercfg-hxfsf\"" Nov 02 17:59:23.101800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.097805 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f07b5b\"" Nov 02 17:59:23.101800 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.099948 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss"] Nov 02 17:59:23.115856 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.111770 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.115856 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.111844 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg258\" (UniqueName: \"kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.200273 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.200235 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz"] Nov 02 17:59:23.222633 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.214208 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.223073 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.223040 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg258\" (UniqueName: \"kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.235250 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.235110 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.239824 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.239752 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.264631 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.264593 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz"] Nov 02 17:59:23.295843 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.295812 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg258\" (UniqueName: \"kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.324717 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.324679 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5796\" (UniqueName: \"kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796\") pod \"payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.324938 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.324912 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.336430 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.336399 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:23.427882 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.427847 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5796\" (UniqueName: \"kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796\") pod \"payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.428351 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.427984 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.428884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.428863 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.485801 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.485768 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5796\" (UniqueName: \"kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796\") pod \"payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.578543 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.578456 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:23.711693 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.711661 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss"] Nov 02 17:59:23.712924 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:59:23.712891 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb401a53_d003_4c5e_84bf_224f48596f31.slice/crio-39596554cf30ac24228b15fe507ef95b71e205fe409d1e6d0a19a7ec33c9699e WatchSource:0}: Error finding container 39596554cf30ac24228b15fe507ef95b71e205fe409d1e6d0a19a7ec33c9699e: Status 404 returned error can't find the container with id 39596554cf30ac24228b15fe507ef95b71e205fe409d1e6d0a19a7ec33c9699e Nov 02 17:59:23.956372 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:23.956345 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz"] Nov 02 17:59:23.958333 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:59:23.958303 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9899c816_f442_49df_b19e_dc9d419d1718.slice/crio-95f9494d1c2c4df71e2e3e410dd51b34e8a0f4f83c32fa97719fe022cd79fdb5 WatchSource:0}: Error finding container 95f9494d1c2c4df71e2e3e410dd51b34e8a0f4f83c32fa97719fe022cd79fdb5: Status 404 returned error can't find the container with id 95f9494d1c2c4df71e2e3e410dd51b34e8a0f4f83c32fa97719fe022cd79fdb5 Nov 02 17:59:24.394718 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.394678 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" event={"ID":"9899c816-f442-49df-b19e-dc9d419d1718","Type":"ContainerStarted","Data":"e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9"} Nov 02 17:59:24.394937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.394752 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" event={"ID":"9899c816-f442-49df-b19e-dc9d419d1718","Type":"ContainerStarted","Data":"95f9494d1c2c4df71e2e3e410dd51b34e8a0f4f83c32fa97719fe022cd79fdb5"} Nov 02 17:59:24.394937 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.394781 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:24.396485 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.396461 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" event={"ID":"fb401a53-d003-4c5e-84bf-224f48596f31","Type":"ContainerStarted","Data":"dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901"} Nov 02 17:59:24.396596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.396498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" event={"ID":"fb401a53-d003-4c5e-84bf-224f48596f31","Type":"ContainerStarted","Data":"39596554cf30ac24228b15fe507ef95b71e205fe409d1e6d0a19a7ec33c9699e"} Nov 02 17:59:24.396596 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.396524 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:24.514192 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:24.514135 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" podStartSLOduration=1.514120525 podStartE2EDuration="1.514120525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:24.512718202 +0000 UTC m=+2151.586245408" watchObservedRunningTime="2026-11-02 17:59:24.514120525 +0000 UTC m=+2151.587647727" Nov 02 17:59:35.406670 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:35.406636 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 17:59:35.407191 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:35.406712 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 17:59:35.524936 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:35.524884 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" podStartSLOduration=13.524870449 podStartE2EDuration="13.524870449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:24.64405744 +0000 UTC m=+2151.717584661" watchObservedRunningTime="2026-11-02 17:59:35.524870449 +0000 UTC m=+2162.598397652" Nov 02 17:59:35.948544 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:35.948503 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 17:59:36.332320 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.332240 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d"] Nov 02 17:59:36.337630 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.337612 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.401599 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.401559 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.401599 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.401599 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvslp\" (UniqueName: \"kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.401840 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.401764 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.401840 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.401796 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.442014 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.441967 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-fd04fa-serving-cert\"" Nov 02 17:59:36.444512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.444489 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d"] Nov 02 17:59:36.502712 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.502680 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.502919 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.502770 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.502919 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:36.502900 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-serving-cert: secret "maas-api-e2e-praxis-a-fd04fa-serving-cert" not found Nov 02 17:59:36.503006 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.502938 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.503006 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:36.502962 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls podName:79b91a83-b6fa-4680-b84f-11366154d5a6 nodeName:}" failed. No retries permitted until 2026-11-02 17:59:37.002945865 +0000 UTC m=+2164.076473049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls") pod "maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" (UID: "79b91a83-b6fa-4680-b84f-11366154d5a6") : secret "maas-api-e2e-praxis-a-fd04fa-serving-cert" not found Nov 02 17:59:36.503006 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.502992 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvslp\" (UniqueName: \"kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.503583 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.503560 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.505564 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.505541 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:36.561625 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:36.561596 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvslp\" (UniqueName: \"kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:37.009135 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:37.009092 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:37.011709 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:37.011681 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:37.251472 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:37.251441 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:37.650300 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:59:37.650267 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b91a83_b6fa_4680_b84f_11366154d5a6.slice/crio-cac51645ae88ce69f89eff99ca1df41214b5c8516de141db5e5e556d97d456e6 WatchSource:0}: Error finding container cac51645ae88ce69f89eff99ca1df41214b5c8516de141db5e5e556d97d456e6: Status 404 returned error can't find the container with id cac51645ae88ce69f89eff99ca1df41214b5c8516de141db5e5e556d97d456e6 Nov 02 17:59:37.650641 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:37.650544 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d"] Nov 02 17:59:38.500884 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:38.500594 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" event={"ID":"79b91a83-b6fa-4680-b84f-11366154d5a6","Type":"ContainerStarted","Data":"cac51645ae88ce69f89eff99ca1df41214b5c8516de141db5e5e556d97d456e6"} Nov 02 17:59:39.507512 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:39.507470 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" event={"ID":"79b91a83-b6fa-4680-b84f-11366154d5a6","Type":"ContainerStarted","Data":"ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a"} Nov 02 17:59:39.507995 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:39.507546 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:39.625668 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:39.625617 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" podStartSLOduration=2.832062583 podStartE2EDuration="3.625600448s" totalImagesPullingTime="793.537865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:39.623999714 +0000 UTC m=+2166.697526915" watchObservedRunningTime="2026-11-02 17:59:39.625600448 +0000 UTC m=+2166.699127691" Nov 02 17:59:45.519427 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:45.519348 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 17:59:52.550016 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:52.549978 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 17:59:55.830836 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.830795 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2"] Nov 02 17:59:55.835657 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.835635 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.871251 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.871214 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rlk\" (UniqueName: \"kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.871439 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.871418 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.871486 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.871460 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.871553 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.871539 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.876591 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.876563 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2"] Nov 02 17:59:55.927764 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.926682 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f42381-serving-cert\"" Nov 02 17:59:55.973638 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.973586 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rlk\" (UniqueName: \"kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.973869 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.973834 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.973960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.973881 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.973960 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.973940 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.974243 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:55.974219 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-serving-cert: secret "maas-api-e2e-praxis-b-f42381-serving-cert" not found Nov 02 17:59:55.974365 ip-10-0-128-71 kubenswrapper[5647]: E1102 17:59:55.974349 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls podName:5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95 nodeName:}" failed. No retries permitted until 2026-11-02 17:59:56.474323767 +0000 UTC m=+2183.547850965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls") pod "maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" (UID: "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95") : secret "maas-api-e2e-praxis-b-f42381-serving-cert" not found Nov 02 17:59:55.975167 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.975141 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:55.977460 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:55.977435 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:56.033561 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:56.033525 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rlk\" (UniqueName: \"kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:56.480814 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:56.480712 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:56.483368 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:56.483337 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:56.750172 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:56.750075 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:57.149463 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:57.149435 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2"] Nov 02 17:59:57.150895 ip-10-0-128-71 kubenswrapper[5647]: W1102 17:59:57.150868 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdf24cc_ed90_4f7c_a8f1_6bb144d71d95.slice/crio-29e4a458f61ba01f6b01f83086d5a52778cae5367eb4f4e2b9ee403e9d1099eb WatchSource:0}: Error finding container 29e4a458f61ba01f6b01f83086d5a52778cae5367eb4f4e2b9ee403e9d1099eb: Status 404 returned error can't find the container with id 29e4a458f61ba01f6b01f83086d5a52778cae5367eb4f4e2b9ee403e9d1099eb Nov 02 17:59:57.643790 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:57.643729 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" event={"ID":"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95","Type":"ContainerStarted","Data":"29e4a458f61ba01f6b01f83086d5a52778cae5367eb4f4e2b9ee403e9d1099eb"} Nov 02 17:59:58.652300 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:58.652259 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" event={"ID":"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95","Type":"ContainerStarted","Data":"0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52"} Nov 02 17:59:58.652722 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:58.652356 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 17:59:58.791721 ip-10-0-128-71 kubenswrapper[5647]: I1102 17:59:58.791656 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" podStartSLOduration=2.94695873 podStartE2EDuration="3.791637981s" totalImagesPullingTime="844.679251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:58.790075288 +0000 UTC m=+2185.863602491" watchObservedRunningTime="2026-11-02 17:59:58.791637981 +0000 UTC m=+2185.865165220" Nov 02 18:00:00.205707 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.205668 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd"] Nov 02 18:00:00.210590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.210565 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.218817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.218775 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b"] Nov 02 18:00:00.223358 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.223333 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.234560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.234528 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2"] Nov 02 18:00:00.239217 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.239182 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.333163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.333131 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:00:00.337944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.337854 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd"] Nov 02 18:00:00.341701 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.341669 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b"] Nov 02 18:00:00.356180 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.356137 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddmvk\" (UniqueName: \"kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk\") pod \"maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.356376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.356191 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ntlt\" (UniqueName: \"kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt\") pod \"maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.356376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.356325 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8jr\" (UniqueName: \"kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr\") pod \"maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.356502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.356446 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.356562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.356542 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.356644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.356631 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.396770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.396709 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2"] Nov 02 18:00:00.457554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.457473 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.457724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.457578 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.457724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.457610 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.457724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.457678 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddmvk\" (UniqueName: \"kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk\") pod \"maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.457724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.457698 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ntlt\" (UniqueName: \"kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt\") pod \"maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.457724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.457744 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8jr\" (UniqueName: \"kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr\") pod \"maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.458566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.458546 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.459119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.459071 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.462775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.459459 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.516890 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.516849 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8jr\" (UniqueName: \"kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr\") pod \"maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.517780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.517693 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddmvk\" (UniqueName: \"kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk\") pod \"maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.517935 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.517798 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ntlt\" (UniqueName: \"kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt\") pod \"maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.526605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.526384 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:00.539749 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.539695 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:00.553139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.553107 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:00.931194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.931161 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd"] Nov 02 18:00:00.932425 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:00:00.932398 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa9c3a4_2e34_4f5c_92c6_5fdb73937842.slice/crio-d488e6223bc71c59c57434a645835fdd16f068060694efbad331e9312ac3d89d WatchSource:0}: Error finding container d488e6223bc71c59c57434a645835fdd16f068060694efbad331e9312ac3d89d: Status 404 returned error can't find the container with id d488e6223bc71c59c57434a645835fdd16f068060694efbad331e9312ac3d89d Nov 02 18:00:00.980543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:00.980500 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2"] Nov 02 18:00:00.984931 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:00:00.984888 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4cb11c_63ac_4f2f_9811_77f0b6eba6c6.slice/crio-3ba581d11d9caa8e38cb9b106486ad89fe7bd23ffd44bef070d99c1055ae0ab6 WatchSource:0}: Error finding container 3ba581d11d9caa8e38cb9b106486ad89fe7bd23ffd44bef070d99c1055ae0ab6: Status 404 returned error can't find the container with id 3ba581d11d9caa8e38cb9b106486ad89fe7bd23ffd44bef070d99c1055ae0ab6 Nov 02 18:00:01.027314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.027063 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b"] Nov 02 18:00:01.030805 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:00:01.030775 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aace8ae_e818_4835_b55c_3b1201f45053.slice/crio-e371f45642b01a0694c9e5113319e6e0f488330805605f4b3ac0a6df9813d4ba WatchSource:0}: Error finding container e371f45642b01a0694c9e5113319e6e0f488330805605f4b3ac0a6df9813d4ba: Status 404 returned error can't find the container with id e371f45642b01a0694c9e5113319e6e0f488330805605f4b3ac0a6df9813d4ba Nov 02 18:00:01.680261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.680223 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerStarted","Data":"54325cdff2602a811e8c20f1b04c65e35ffdca3e558cfea49320398545481e22"} Nov 02 18:00:01.680800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.680293 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerStarted","Data":"d488e6223bc71c59c57434a645835fdd16f068060694efbad331e9312ac3d89d"} Nov 02 18:00:01.682088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.682056 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerStarted","Data":"b0c18de72a54d75b6e3e31459be38a3b45ccff63cafbf082e03febdc6261655e"} Nov 02 18:00:01.682213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.682110 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerStarted","Data":"e371f45642b01a0694c9e5113319e6e0f488330805605f4b3ac0a6df9813d4ba"} Nov 02 18:00:01.683772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.683716 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerStarted","Data":"647f99a6472f105981c73a59633002c2e099fbf7cfa42050922f598ce05e6eb4"} Nov 02 18:00:01.683893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.683795 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerStarted","Data":"3ba581d11d9caa8e38cb9b106486ad89fe7bd23ffd44bef070d99c1055ae0ab6"} Nov 02 18:00:01.797284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.797233 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" podStartSLOduration=1.79721908 podStartE2EDuration="1.79721908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.79650533 +0000 UTC m=+2188.870032533" watchObservedRunningTime="2026-11-02 18:00:01.79721908 +0000 UTC m=+2188.870746284" Nov 02 18:00:01.915045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:01.914993 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" podStartSLOduration=1.91497839 podStartE2EDuration="1.91497839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:01.912114929 +0000 UTC m=+2188.985642131" watchObservedRunningTime="2026-11-02 18:00:01.91497839 +0000 UTC m=+2188.988505593" Nov 02 18:00:02.078880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.078822 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" podStartSLOduration=2.078806979 podStartE2EDuration="2.078806979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:02.076145128 +0000 UTC m=+2189.149672331" watchObservedRunningTime="2026-11-02 18:00:02.078806979 +0000 UTC m=+2189.152334215" Nov 02 18:00:02.694008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.693970 5647 generic.go:356] "Generic (PLEG): container finished" podID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerID="54325cdff2602a811e8c20f1b04c65e35ffdca3e558cfea49320398545481e22" exitCode=7 Nov 02 18:00:02.694478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.694052 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerDied","Data":"54325cdff2602a811e8c20f1b04c65e35ffdca3e558cfea49320398545481e22"} Nov 02 18:00:02.694478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.694315 5647 scope.go:122] "RemoveContainer" containerID="54325cdff2602a811e8c20f1b04c65e35ffdca3e558cfea49320398545481e22" Nov 02 18:00:02.696030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.696003 5647 generic.go:356] "Generic (PLEG): container finished" podID="3aace8ae-e818-4835-b55c-3b1201f45053" containerID="b0c18de72a54d75b6e3e31459be38a3b45ccff63cafbf082e03febdc6261655e" exitCode=60 Nov 02 18:00:02.696169 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.696098 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerDied","Data":"b0c18de72a54d75b6e3e31459be38a3b45ccff63cafbf082e03febdc6261655e"} Nov 02 18:00:02.696317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.696307 5647 scope.go:122] "RemoveContainer" containerID="b0c18de72a54d75b6e3e31459be38a3b45ccff63cafbf082e03febdc6261655e" Nov 02 18:00:02.698193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.698170 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerID="647f99a6472f105981c73a59633002c2e099fbf7cfa42050922f598ce05e6eb4" exitCode=60 Nov 02 18:00:02.698303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.698218 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerDied","Data":"647f99a6472f105981c73a59633002c2e099fbf7cfa42050922f598ce05e6eb4"} Nov 02 18:00:02.698578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:02.698557 5647 scope.go:122] "RemoveContainer" containerID="647f99a6472f105981c73a59633002c2e099fbf7cfa42050922f598ce05e6eb4" Nov 02 18:00:03.706688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.706652 5647 generic.go:356] "Generic (PLEG): container finished" podID="3aace8ae-e818-4835-b55c-3b1201f45053" containerID="2109de95533697d738e593bb678461f291228bb564a64d001ee63d6fa9dd1f3c" exitCode=60 Nov 02 18:00:03.707221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.706755 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerDied","Data":"2109de95533697d738e593bb678461f291228bb564a64d001ee63d6fa9dd1f3c"} Nov 02 18:00:03.707221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.706817 5647 scope.go:122] "RemoveContainer" containerID="b0c18de72a54d75b6e3e31459be38a3b45ccff63cafbf082e03febdc6261655e" Nov 02 18:00:03.707221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.707092 5647 scope.go:122] "RemoveContainer" containerID="2109de95533697d738e593bb678461f291228bb564a64d001ee63d6fa9dd1f3c" Nov 02 18:00:03.707435 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:03.707410 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b_odh-ai-gateway-infra(3aace8ae-e818-4835-b55c-3b1201f45053)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" Nov 02 18:00:03.708621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.708596 5647 generic.go:356] "Generic (PLEG): container finished" podID="531cd99d-2559-464f-899d-88a9f483efe2" containerID="ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c" exitCode=0 Nov 02 18:00:03.708782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.708668 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" event={"ID":"531cd99d-2559-464f-899d-88a9f483efe2","Type":"ContainerDied","Data":"ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c"} Nov 02 18:00:03.709104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.709087 5647 scope.go:122] "RemoveContainer" containerID="ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c" Nov 02 18:00:03.710677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.710644 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:00:03.710805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.710726 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:00:03.710869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.710815 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerStarted","Data":"9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81"} Nov 02 18:00:03.710869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.710830 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:00:03.713127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.713101 5647 generic.go:356] "Generic (PLEG): container finished" podID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerID="0e183c613944b568aa894e817fcc74ef53ce4e5d6550d27f0de8184b01a6d8f4" exitCode=7 Nov 02 18:00:03.713262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.713156 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerDied","Data":"0e183c613944b568aa894e817fcc74ef53ce4e5d6550d27f0de8184b01a6d8f4"} Nov 02 18:00:03.713371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.713357 5647 scope.go:122] "RemoveContainer" containerID="0e183c613944b568aa894e817fcc74ef53ce4e5d6550d27f0de8184b01a6d8f4" Nov 02 18:00:03.713594 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:03.713552 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd_odh-ai-gateway-infra(3fa9c3a4-2e34-4f5c-92c6-5fdb73937842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" Nov 02 18:00:03.723240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.723222 5647 scope.go:122] "RemoveContainer" containerID="54325cdff2602a811e8c20f1b04c65e35ffdca3e558cfea49320398545481e22" Nov 02 18:00:03.995825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:03.995785 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:00:04.664418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.664393 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 18:00:04.734654 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.734614 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" event={"ID":"531cd99d-2559-464f-899d-88a9f483efe2","Type":"ContainerStarted","Data":"11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca"} Nov 02 18:00:04.736624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.736596 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerID="9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81" exitCode=60 Nov 02 18:00:04.736791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.736683 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerDied","Data":"9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81"} Nov 02 18:00:04.736791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.736760 5647 scope.go:122] "RemoveContainer" containerID="647f99a6472f105981c73a59633002c2e099fbf7cfa42050922f598ce05e6eb4" Nov 02 18:00:04.736929 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.736910 5647 scope.go:122] "RemoveContainer" containerID="9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81" Nov 02 18:00:04.737225 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:04.737198 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2_odh-ai-gateway-infra(1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" Nov 02 18:00:04.738829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.738807 5647 scope.go:122] "RemoveContainer" containerID="0e183c613944b568aa894e817fcc74ef53ce4e5d6550d27f0de8184b01a6d8f4" Nov 02 18:00:04.739088 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:04.739065 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd_odh-ai-gateway-infra(3fa9c3a4-2e34-4f5c-92c6-5fdb73937842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" Nov 02 18:00:04.740440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.740425 5647 scope.go:122] "RemoveContainer" containerID="2109de95533697d738e593bb678461f291228bb564a64d001ee63d6fa9dd1f3c" Nov 02 18:00:04.740642 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:04.740620 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b_odh-ai-gateway-infra(3aace8ae-e818-4835-b55c-3b1201f45053)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" Nov 02 18:00:04.773003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.772973 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:00:04.774307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.774287 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:00:04.774360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:04.774333 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:00:05.748544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:05.748516 5647 scope.go:122] "RemoveContainer" containerID="9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81" Nov 02 18:00:05.749077 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:05.748902 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2_odh-ai-gateway-infra(1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" Nov 02 18:00:05.777187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:05.777148 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:00:05.777445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:05.777418 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:00:05.953327 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:05.953290 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:00:06.773470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:06.773434 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:00:06.774019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:06.773498 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:00:07.773648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:07.773612 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:00:07.774105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:07.773668 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:00:08.773233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:08.773196 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:00:08.773440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:08.773253 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:00:09.773053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:09.773018 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Nov 02 18:00:09.773633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:09.773072 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Nov 02 18:00:10.777834 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:10.777795 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:00:10.778315 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:10.778146 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:00:10.779436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:10.779273 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:00:17.074870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:17.074839 5647 scope.go:122] "RemoveContainer" containerID="9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81" Nov 02 18:00:17.832810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:17.832692 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerStarted","Data":"94e70477fb7f4b9eb9bf4e760d1635beb274fd2b5676833774fb375a9d6259d5"} Nov 02 18:00:18.074051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.074023 5647 scope.go:122] "RemoveContainer" containerID="0e183c613944b568aa894e817fcc74ef53ce4e5d6550d27f0de8184b01a6d8f4" Nov 02 18:00:18.074253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.074125 5647 scope.go:122] "RemoveContainer" containerID="2109de95533697d738e593bb678461f291228bb564a64d001ee63d6fa9dd1f3c" Nov 02 18:00:18.840498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.840443 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerStarted","Data":"0afa9366b61afbe70d2d8ce4d5c1ef6b753e24a1ed717a06a8e940e852ab0509"} Nov 02 18:00:18.842341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.842314 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerStarted","Data":"eca406e8c2a760f01fcedf1aa163cc4b48f7fbbdf5b6e41367800df7c24bb976"} Nov 02 18:00:18.844228 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.844202 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerID="94e70477fb7f4b9eb9bf4e760d1635beb274fd2b5676833774fb375a9d6259d5" exitCode=60 Nov 02 18:00:18.844372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.844266 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerDied","Data":"94e70477fb7f4b9eb9bf4e760d1635beb274fd2b5676833774fb375a9d6259d5"} Nov 02 18:00:18.844372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.844317 5647 scope.go:122] "RemoveContainer" containerID="9517c75a40f36854675593e78a32ed82540228a6fb7b0c073b68ea2d4a3e0c81" Nov 02 18:00:18.844678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.844659 5647 scope.go:122] "RemoveContainer" containerID="94e70477fb7f4b9eb9bf4e760d1635beb274fd2b5676833774fb375a9d6259d5" Nov 02 18:00:18.844987 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:18.844962 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2_odh-ai-gateway-infra(1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" Nov 02 18:00:18.982297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:18.982264 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2"] Nov 02 18:00:19.929339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.929284 5647 generic.go:356] "Generic (PLEG): container finished" podID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerID="0afa9366b61afbe70d2d8ce4d5c1ef6b753e24a1ed717a06a8e940e852ab0509" exitCode=60 Nov 02 18:00:19.930064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.930022 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerDied","Data":"0afa9366b61afbe70d2d8ce4d5c1ef6b753e24a1ed717a06a8e940e852ab0509"} Nov 02 18:00:19.930314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.930296 5647 scope.go:122] "RemoveContainer" containerID="0e183c613944b568aa894e817fcc74ef53ce4e5d6550d27f0de8184b01a6d8f4" Nov 02 18:00:19.932330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.932292 5647 scope.go:122] "RemoveContainer" containerID="0afa9366b61afbe70d2d8ce4d5c1ef6b753e24a1ed717a06a8e940e852ab0509" Nov 02 18:00:19.933035 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:19.933009 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd_odh-ai-gateway-infra(3fa9c3a4-2e34-4f5c-92c6-5fdb73937842)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" Nov 02 18:00:19.939639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.939603 5647 generic.go:356] "Generic (PLEG): container finished" podID="3aace8ae-e818-4835-b55c-3b1201f45053" containerID="eca406e8c2a760f01fcedf1aa163cc4b48f7fbbdf5b6e41367800df7c24bb976" exitCode=60 Nov 02 18:00:19.940101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.939887 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerDied","Data":"eca406e8c2a760f01fcedf1aa163cc4b48f7fbbdf5b6e41367800df7c24bb976"} Nov 02 18:00:19.943604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.943581 5647 scope.go:122] "RemoveContainer" containerID="eca406e8c2a760f01fcedf1aa163cc4b48f7fbbdf5b6e41367800df7c24bb976" Nov 02 18:00:19.944197 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:19.944161 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b_odh-ai-gateway-infra(3aace8ae-e818-4835-b55c-3b1201f45053)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" Nov 02 18:00:19.963132 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:19.963104 5647 scope.go:122] "RemoveContainer" containerID="2109de95533697d738e593bb678461f291228bb564a64d001ee63d6fa9dd1f3c" Nov 02 18:00:20.001983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.001863 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd"] Nov 02 18:00:20.118660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.118628 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:20.198791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.198686 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk\" (UniqueName: \"kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk\") pod \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " Nov 02 18:00:20.198791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.198761 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca\") pod \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\" (UID: \"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6\") " Nov 02 18:00:20.199128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.199103 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca" pod "1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" (UID: "1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:20.199433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.199414 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:00:20.201136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.201111 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk" pod "1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" (UID: "1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6"). InnerVolumeSpecName "kube-api-access-ddmvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:20.208032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.208006 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b"] Nov 02 18:00:20.300748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.300698 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddmvk\" (UniqueName: \"kubernetes.io/projected/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6-kube-api-access-ddmvk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:00:20.947478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.947451 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" Nov 02 18:00:20.947987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.947492 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2" event={"ID":"1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6","Type":"ContainerDied","Data":"3ba581d11d9caa8e38cb9b106486ad89fe7bd23ffd44bef070d99c1055ae0ab6"} Nov 02 18:00:20.947987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:20.947560 5647 scope.go:122] "RemoveContainer" containerID="94e70477fb7f4b9eb9bf4e760d1635beb274fd2b5676833774fb375a9d6259d5" Nov 02 18:00:21.172294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.172148 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:21.193126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.193088 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:21.242560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.242459 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca\") pod \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " Nov 02 18:00:21.242560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.242484 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2"] Nov 02 18:00:21.242560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.242529 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr\" (UniqueName: \"kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr\") pod \"3aace8ae-e818-4835-b55c-3b1201f45053\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " Nov 02 18:00:21.242930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.242620 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt\" (UniqueName: \"kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt\") pod \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\" (UID: \"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842\") " Nov 02 18:00:21.242930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.242660 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca\") pod \"3aace8ae-e818-4835-b55c-3b1201f45053\" (UID: \"3aace8ae-e818-4835-b55c-3b1201f45053\") " Nov 02 18:00:21.243038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.242910 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca" pod "3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" (UID: "3fa9c3a4-2e34-4f5c-92c6-5fdb73937842"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:21.243103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.243081 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca" pod "3aace8ae-e818-4835-b55c-3b1201f45053" (UID: "3aace8ae-e818-4835-b55c-3b1201f45053"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:00:21.243146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.243135 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:00:21.244990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.244966 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr" pod "3aace8ae-e818-4835-b55c-3b1201f45053" (UID: "3aace8ae-e818-4835-b55c-3b1201f45053"). InnerVolumeSpecName "kube-api-access-5t8jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:21.245083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.245002 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt" pod "3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" (UID: "3fa9c3a4-2e34-4f5c-92c6-5fdb73937842"). InnerVolumeSpecName "kube-api-access-8ntlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:00:21.251096 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.251071 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-fd04fa-29894040-7rqj2"] Nov 02 18:00:21.343934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.343896 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8jr\" (UniqueName: \"kubernetes.io/projected/3aace8ae-e818-4835-b55c-3b1201f45053-kube-api-access-5t8jr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:00:21.343934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.343931 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ntlt\" (UniqueName: \"kubernetes.io/projected/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842-kube-api-access-8ntlt\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:00:21.343934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.343941 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aace8ae-e818-4835-b55c-3b1201f45053-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:00:21.963925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.963894 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" Nov 02 18:00:21.963925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.963911 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd" event={"ID":"3fa9c3a4-2e34-4f5c-92c6-5fdb73937842","Type":"ContainerDied","Data":"d488e6223bc71c59c57434a645835fdd16f068060694efbad331e9312ac3d89d"} Nov 02 18:00:21.964501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.963987 5647 scope.go:122] "RemoveContainer" containerID="0afa9366b61afbe70d2d8ce4d5c1ef6b753e24a1ed717a06a8e940e852ab0509" Nov 02 18:00:21.965445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.965424 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" event={"ID":"3aace8ae-e818-4835-b55c-3b1201f45053","Type":"ContainerDied","Data":"e371f45642b01a0694c9e5113319e6e0f488330805605f4b3ac0a6df9813d4ba"} Nov 02 18:00:21.965542 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.965467 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b" Nov 02 18:00:21.975524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:21.975452 5647 scope.go:122] "RemoveContainer" containerID="eca406e8c2a760f01fcedf1aa163cc4b48f7fbbdf5b6e41367800df7c24bb976" Nov 02 18:00:22.138488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:22.138450 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b"] Nov 02 18:00:22.150378 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:22.150337 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-5a1e27-29894040-9fn4b"] Nov 02 18:00:22.286768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:22.286659 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd"] Nov 02 18:00:22.292847 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:22.292813 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-f42381-29894040-9z2dd"] Nov 02 18:00:23.082823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:23.082784 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" path="/var/lib/kubelet/pods/1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6/volumes" Nov 02 18:00:23.083387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:23.083364 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" path="/var/lib/kubelet/pods/3aace8ae-e818-4835-b55c-3b1201f45053/volumes" Nov 02 18:00:23.084485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:23.084453 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" path="/var/lib/kubelet/pods/3fa9c3a4-2e34-4f5c-92c6-5fdb73937842/volumes" Nov 02 18:00:31.027786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:31.027726 5647 generic.go:356] "Generic (PLEG): container finished" podID="89fd7905-7063-46bf-9e1f-1d7755094293" containerID="938e9e2a92eb3e4029809a35c41949c4c9a9db74c63bdcc6cb30c9a1aa7d6092" exitCode=0 Nov 02 18:00:31.028260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:31.027795 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" event={"ID":"89fd7905-7063-46bf-9e1f-1d7755094293","Type":"ContainerDied","Data":"938e9e2a92eb3e4029809a35c41949c4c9a9db74c63bdcc6cb30c9a1aa7d6092"} Nov 02 18:00:31.028260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:31.028206 5647 scope.go:122] "RemoveContainer" containerID="938e9e2a92eb3e4029809a35c41949c4c9a9db74c63bdcc6cb30c9a1aa7d6092" Nov 02 18:00:31.029851 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:31.029817 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:00:31.029919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:31.029892 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:00:31.029956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:31.029927 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:00:32.037431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:32.037395 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" event={"ID":"89fd7905-7063-46bf-9e1f-1d7755094293","Type":"ContainerStarted","Data":"ecf9ce106d685fa08be405822040958bf55a4b4ddd9dafdb3ad8f311a614dca9"} Nov 02 18:00:32.574438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:32.574394 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 18:00:32.575776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:32.575717 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:00:32.575910 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:32.575804 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 18:00:33.573780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:33.573712 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:00:33.574272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:33.573804 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 18:00:34.573850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:34.573806 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:00:34.574259 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:34.573886 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 18:00:35.574331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:35.574293 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:00:35.574822 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:35.574356 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 18:00:35.953626 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:00:35.953588 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:00:36.165030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:36.165004 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 18:00:36.573937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:36.573904 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 02 18:00:36.574133 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:36.573959 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 02 18:00:37.580326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:37.580293 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 18:00:37.581670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:37.581650 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 18:00:40.066346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:00:40.066305 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:01:05.947257 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:01:05.947212 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:01:07.832326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:07.832285 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:01:09.010328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.010291 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29"] Nov 02 18:01:09.011879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.011853 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:01:09.013034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013013 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:01:09.013182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013172 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:01:09.013243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013236 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:01:09.013339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013332 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:01:09.013391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013384 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:01:09.013446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013440 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.013497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013490 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.013566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013559 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.013621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013614 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.013674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013668 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.013724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013717 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.013813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013805 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:01:09.013868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.013861 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:01:09.014044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014035 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:01:09.014113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014107 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:01:09.014165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014159 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.014223 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014217 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.014272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014266 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:01:09.014324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014318 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aace8ae-e818-4835-b55c-3b1201f45053" containerName="cleanup" Nov 02 18:01:09.014375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.014369 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:01:09.018506 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.018486 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.169766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.169159 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cf2b8a\"" Nov 02 18:01:09.169766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.169245 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cf2b8a-dockercfg-q5hxt\"" Nov 02 18:01:09.207939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.207896 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29"] Nov 02 18:01:09.219999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.219541 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.219999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.219813 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl82l\" (UniqueName: \"kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l\") pod \"payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.320777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.320660 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.320961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.320870 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl82l\" (UniqueName: \"kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l\") pod \"payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.321381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.321360 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.379570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.379535 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl82l\" (UniqueName: \"kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l\") pod \"payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:09.637918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:09.637879 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:10.030833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:10.030788 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29"] Nov 02 18:01:10.034344 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:01:10.034308 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d1322d_d323_4c13_b50f_5cbbca395367.slice/crio-9f5d282b80fed5b8d654759f4d37faee8e5354ade8a0863aa34acbbce64bb115 WatchSource:0}: Error finding container 9f5d282b80fed5b8d654759f4d37faee8e5354ade8a0863aa34acbbce64bb115: Status 404 returned error can't find the container with id 9f5d282b80fed5b8d654759f4d37faee8e5354ade8a0863aa34acbbce64bb115 Nov 02 18:01:10.297502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:10.297412 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" event={"ID":"45d1322d-d323-4c13-b50f-5cbbca395367","Type":"ContainerStarted","Data":"139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a"} Nov 02 18:01:10.297502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:10.297462 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" event={"ID":"45d1322d-d323-4c13-b50f-5cbbca395367","Type":"ContainerStarted","Data":"9f5d282b80fed5b8d654759f4d37faee8e5354ade8a0863aa34acbbce64bb115"} Nov 02 18:01:10.297777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:10.297630 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:10.419759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:10.419227 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" podStartSLOduration=2.419201113 podStartE2EDuration="2.419201113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:10.415293892 +0000 UTC m=+2257.488821115" watchObservedRunningTime="2026-11-02 18:01:10.419201113 +0000 UTC m=+2257.492728318" Nov 02 18:01:19.383154 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:19.383118 5647 generic.go:356] "Generic (PLEG): container finished" podID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerID="f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9" exitCode=0 Nov 02 18:01:19.383758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:19.383202 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" event={"ID":"52cd3067-1b2c-492c-a7d8-e28d39754a5a","Type":"ContainerDied","Data":"f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9"} Nov 02 18:01:19.383758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:19.383722 5647 scope.go:122] "RemoveContainer" containerID="f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9" Nov 02 18:01:19.385393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:19.385357 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:01:19.385512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:19.385441 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:01:19.385512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:19.385491 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:01:20.391992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:20.391960 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" event={"ID":"52cd3067-1b2c-492c-a7d8-e28d39754a5a","Type":"ContainerStarted","Data":"d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d"} Nov 02 18:01:20.439959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:20.439916 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 18:01:20.440245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:20.440219 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:20.440293 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:20.440271 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 18:01:21.352112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:21.352078 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:01:21.439425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:21.439387 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:21.439873 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:21.439441 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 18:01:22.440418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:22.440385 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:22.440870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:22.440437 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 18:01:23.443092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:23.443031 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:23.443648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:23.443230 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 18:01:23.447346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:23.447314 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd"] Nov 02 18:01:24.440278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:24.440244 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:24.440485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:24.440312 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 18:01:25.439891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:25.439855 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 02 18:01:25.440312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:25.439939 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 02 18:01:26.444836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:26.444805 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 18:01:26.445223 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:26.445086 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 18:01:26.445763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:26.445745 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 18:01:28.448483 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.448421 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" podUID="da3fab43-cd7c-486d-bc78-6c1616a78a22" containerName="payload-pre-processing" containerID="cri-o://f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67" gracePeriod=25 Nov 02 18:01:28.720433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.720408 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 18:01:28.814727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.814693 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume\") pod \"da3fab43-cd7c-486d-bc78-6c1616a78a22\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " Nov 02 18:01:28.814929 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.814816 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j\" (UniqueName: \"kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j\") pod \"da3fab43-cd7c-486d-bc78-6c1616a78a22\" (UID: \"da3fab43-cd7c-486d-bc78-6c1616a78a22\") " Nov 02 18:01:28.815125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.815102 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume" pod "da3fab43-cd7c-486d-bc78-6c1616a78a22" (UID: "da3fab43-cd7c-486d-bc78-6c1616a78a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:28.815425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.815408 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da3fab43-cd7c-486d-bc78-6c1616a78a22-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:01:28.817075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.817042 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j" pod "da3fab43-cd7c-486d-bc78-6c1616a78a22" (UID: "da3fab43-cd7c-486d-bc78-6c1616a78a22"). InnerVolumeSpecName "kube-api-access-9js5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:28.916424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:28.916390 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9js5j\" (UniqueName: \"kubernetes.io/projected/da3fab43-cd7c-486d-bc78-6c1616a78a22-kube-api-access-9js5j\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:01:29.511780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.511676 5647 generic.go:356] "Generic (PLEG): container finished" podID="da3fab43-cd7c-486d-bc78-6c1616a78a22" containerID="f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67" exitCode=0 Nov 02 18:01:29.511780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.511724 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" Nov 02 18:01:29.511780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.511755 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" event={"ID":"da3fab43-cd7c-486d-bc78-6c1616a78a22","Type":"ContainerDied","Data":"f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67"} Nov 02 18:01:29.512404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.511816 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd" event={"ID":"da3fab43-cd7c-486d-bc78-6c1616a78a22","Type":"ContainerDied","Data":"7272990bfc1182334638ff3a6319a0b02d019ff23219e97076dfb4f819968c25"} Nov 02 18:01:29.512404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.511843 5647 scope.go:122] "RemoveContainer" containerID="f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67" Nov 02 18:01:29.522036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.522015 5647 scope.go:122] "RemoveContainer" containerID="f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67" Nov 02 18:01:29.522327 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:01:29.522299 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67\": container with ID starting with f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67 not found: ID does not exist" containerID="f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67" Nov 02 18:01:29.522454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.522338 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67"} err="failed to get container status \"f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67\": rpc error: code = NotFound desc = could not find container \"f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67\": container with ID starting with f19be2eefd9bb34452b540d2d96e0f576b52220abd4123f70596e909c9132a67 not found: ID does not exist" Nov 02 18:01:29.679664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.679631 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd"] Nov 02 18:01:29.730481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:29.730443 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5a1e27-5496f64f9-8bxnd"] Nov 02 18:01:31.082335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:31.082302 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da3fab43-cd7c-486d-bc78-6c1616a78a22" path="/var/lib/kubelet/pods/da3fab43-cd7c-486d-bc78-6c1616a78a22/volumes" Nov 02 18:01:35.948485 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:01:35.948446 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:01:42.030252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:42.030211 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:01:50.912021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:50.911982 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz"] Nov 02 18:01:51.024799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:51.024762 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss"] Nov 02 18:01:55.912657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:55.912581 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" podUID="9899c816-f442-49df-b19e-dc9d419d1718" containerName="payload-processing" containerID="cri-o://e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9" gracePeriod=25 Nov 02 18:01:56.026040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.025999 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" podUID="fb401a53-d003-4c5e-84bf-224f48596f31" containerName="payload-pre-processing" containerID="cri-o://dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901" gracePeriod=25 Nov 02 18:01:56.183627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.183561 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 18:01:56.293818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.293796 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796\" (UniqueName: \"kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796\") pod \"9899c816-f442-49df-b19e-dc9d419d1718\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " Nov 02 18:01:56.293914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.293893 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume\") pod \"9899c816-f442-49df-b19e-dc9d419d1718\" (UID: \"9899c816-f442-49df-b19e-dc9d419d1718\") " Nov 02 18:01:56.294407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.294367 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume" pod "9899c816-f442-49df-b19e-dc9d419d1718" (UID: "9899c816-f442-49df-b19e-dc9d419d1718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:56.296343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.296318 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796" pod "9899c816-f442-49df-b19e-dc9d419d1718" (UID: "9899c816-f442-49df-b19e-dc9d419d1718"). InnerVolumeSpecName "kube-api-access-f5796". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:56.307296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.307275 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 18:01:56.395128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.395095 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258\" (UniqueName: \"kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258\") pod \"fb401a53-d003-4c5e-84bf-224f48596f31\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " Nov 02 18:01:56.395344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.395168 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume\") pod \"fb401a53-d003-4c5e-84bf-224f48596f31\" (UID: \"fb401a53-d003-4c5e-84bf-224f48596f31\") " Nov 02 18:01:56.395527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.395514 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5796\" (UniqueName: \"kubernetes.io/projected/9899c816-f442-49df-b19e-dc9d419d1718-kube-api-access-f5796\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:01:56.395599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.395529 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899c816-f442-49df-b19e-dc9d419d1718-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:01:56.395599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.395541 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume" pod "fb401a53-d003-4c5e-84bf-224f48596f31" (UID: "fb401a53-d003-4c5e-84bf-224f48596f31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:01:56.397561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.397541 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258" pod "fb401a53-d003-4c5e-84bf-224f48596f31" (UID: "fb401a53-d003-4c5e-84bf-224f48596f31"). InnerVolumeSpecName "kube-api-access-mg258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:01:56.496880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.496797 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg258\" (UniqueName: \"kubernetes.io/projected/fb401a53-d003-4c5e-84bf-224f48596f31-kube-api-access-mg258\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:01:56.496880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.496832 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb401a53-d003-4c5e-84bf-224f48596f31-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:01:56.695505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.695457 5647 generic.go:356] "Generic (PLEG): container finished" podID="fb401a53-d003-4c5e-84bf-224f48596f31" containerID="dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901" exitCode=0 Nov 02 18:01:56.695717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.695526 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" Nov 02 18:01:56.695717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.695538 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" event={"ID":"fb401a53-d003-4c5e-84bf-224f48596f31","Type":"ContainerDied","Data":"dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901"} Nov 02 18:01:56.695717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.695630 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss" event={"ID":"fb401a53-d003-4c5e-84bf-224f48596f31","Type":"ContainerDied","Data":"39596554cf30ac24228b15fe507ef95b71e205fe409d1e6d0a19a7ec33c9699e"} Nov 02 18:01:56.695717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.695664 5647 scope.go:122] "RemoveContainer" containerID="dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901" Nov 02 18:01:56.697355 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.697328 5647 generic.go:356] "Generic (PLEG): container finished" podID="9899c816-f442-49df-b19e-dc9d419d1718" containerID="e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9" exitCode=0 Nov 02 18:01:56.697475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.697385 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" Nov 02 18:01:56.697475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.697399 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" event={"ID":"9899c816-f442-49df-b19e-dc9d419d1718","Type":"ContainerDied","Data":"e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9"} Nov 02 18:01:56.697475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.697443 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz" event={"ID":"9899c816-f442-49df-b19e-dc9d419d1718","Type":"ContainerDied","Data":"95f9494d1c2c4df71e2e3e410dd51b34e8a0f4f83c32fa97719fe022cd79fdb5"} Nov 02 18:01:56.706700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.706680 5647 scope.go:122] "RemoveContainer" containerID="dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901" Nov 02 18:01:56.707150 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:01:56.706992 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901\": container with ID starting with dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901 not found: ID does not exist" containerID="dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901" Nov 02 18:01:56.707150 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.707019 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901"} err="failed to get container status \"dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901\": rpc error: code = NotFound desc = could not find container \"dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901\": container with ID starting with dbec3f52212b213b846c783c0152640d8a5c8394d52d18a2b57654e687092901 not found: ID does not exist" Nov 02 18:01:56.707150 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.707036 5647 scope.go:122] "RemoveContainer" containerID="e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9" Nov 02 18:01:56.716983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.716960 5647 scope.go:122] "RemoveContainer" containerID="e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9" Nov 02 18:01:56.717237 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:01:56.717219 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9\": container with ID starting with e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9 not found: ID does not exist" containerID="e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9" Nov 02 18:01:56.717295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.717244 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9"} err="failed to get container status \"e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9\": rpc error: code = NotFound desc = could not find container \"e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9\": container with ID starting with e576bf06f67abaa3b651bec4873813577dcb59ed88e03ab8e3753f6a96cff8e9 not found: ID does not exist" Nov 02 18:01:56.863695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.863666 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss"] Nov 02 18:01:56.914633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.914602 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-db4c8998c-tttss"] Nov 02 18:01:56.979166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.979135 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz"] Nov 02 18:01:56.985675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:56.985649 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f07b5b-56f5579ff7-qvqfz"] Nov 02 18:01:57.081378 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:57.081346 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9899c816-f442-49df-b19e-dc9d419d1718" path="/var/lib/kubelet/pods/9899c816-f442-49df-b19e-dc9d419d1718/volumes" Nov 02 18:01:57.081769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:01:57.081756 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb401a53-d003-4c5e-84bf-224f48596f31" path="/var/lib/kubelet/pods/fb401a53-d003-4c5e-84bf-224f48596f31/volumes" Nov 02 18:02:05.947909 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:02:05.947867 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:02:06.372218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.372186 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp"] Nov 02 18:02:06.372453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.372411 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" podUID="c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" containerName="maas-api" containerID="cri-o://29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4" gracePeriod=30 Nov 02 18:02:06.672395 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.672371 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 18:02:06.762040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.762007 5647 generic.go:356] "Generic (PLEG): container finished" podID="c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" containerID="29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4" exitCode=0 Nov 02 18:02:06.762254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.762069 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" Nov 02 18:02:06.762254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.762095 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" event={"ID":"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e","Type":"ContainerDied","Data":"29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4"} Nov 02 18:02:06.762254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.762196 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp" event={"ID":"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e","Type":"ContainerDied","Data":"4710f408f630c4c61ac4d685ee0f39a2695824391f5d5ca64909e297919b61d3"} Nov 02 18:02:06.762254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.762230 5647 scope.go:122] "RemoveContainer" containerID="29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4" Nov 02 18:02:06.772855 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.772834 5647 scope.go:122] "RemoveContainer" containerID="29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4" Nov 02 18:02:06.773149 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:02:06.773124 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4\": container with ID starting with 29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4 not found: ID does not exist" containerID="29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4" Nov 02 18:02:06.773206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.773164 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4"} err="failed to get container status \"29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4\": rpc error: code = NotFound desc = could not find container \"29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4\": container with ID starting with 29f830ad0b021ea352d8cc900efac2ad1bf944caf1a43af41809c95f3c8479f4 not found: ID does not exist" Nov 02 18:02:06.835433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.835392 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\") pod \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " Nov 02 18:02:06.835587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.835549 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca\") pod \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " Nov 02 18:02:06.835656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.835637 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv\" (UniqueName: \"kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv\") pod \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " Nov 02 18:02:06.835709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.835670 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs\") pod \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\" (UID: \"c989b4af-ae4f-4bc1-a8e2-8753a19cb76e\") " Nov 02 18:02:06.836039 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.836007 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca" pod "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" (UID: "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:06.836224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.836204 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:06.838125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.838098 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs" pod "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" (UID: "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:06.838311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.838290 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls" pod "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" (UID: "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:06.838394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.838303 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv" pod "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" (UID: "c989b4af-ae4f-4bc1-a8e2-8753a19cb76e"). InnerVolumeSpecName "kube-api-access-chjqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:06.937898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.937815 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chjqv\" (UniqueName: \"kubernetes.io/projected/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-kube-api-access-chjqv\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:06.937898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.937846 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:06.937898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:06.937856 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:07.185046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:07.184998 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp"] Nov 02 18:02:07.188824 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:07.188720 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5a1e27-685684966b-gg4vp"] Nov 02 18:02:09.081180 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:09.081145 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" path="/var/lib/kubelet/pods/c989b4af-ae4f-4bc1-a8e2-8753a19cb76e/volumes" Nov 02 18:02:12.894491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:12.894456 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9"] Nov 02 18:02:12.894950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:12.894655 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" containerID="cri-o://5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90" gracePeriod=30 Nov 02 18:02:18.165248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.165185 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 18:02:18.200705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200669 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.200906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200727 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.200906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200824 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.200906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200862 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.200906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200895 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.201104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200925 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.201104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.200981 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.201104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201056 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.201270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201118 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs\") pod \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\" (UID: \"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1\") " Nov 02 18:02:18.201270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201208 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:18.201387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201218 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:18.201387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201277 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:18.201387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201304 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:18.201559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201516 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:18.201708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201687 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.201786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201718 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.201786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201753 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.201786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201766 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.201786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.201780 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.203321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.203299 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "kube-api-access-pwkmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:18.203434 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.203331 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:18.203835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.203815 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:18.203975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.203812 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo" pod "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" (UID: "dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:02:18.302749 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.302705 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkmm\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-kube-api-access-pwkmm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.302938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.302772 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.302938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.302785 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.302938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.302800 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:18.841626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.841591 5647 generic.go:356] "Generic (PLEG): container finished" podID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerID="5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90" exitCode=0 Nov 02 18:02:18.841843 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.841645 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" Nov 02 18:02:18.841843 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.841667 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" event={"ID":"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1","Type":"ContainerDied","Data":"5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90"} Nov 02 18:02:18.841843 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.841764 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9" event={"ID":"dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1","Type":"ContainerDied","Data":"67511c6a458343354151b4c0a5c34f4afee92c4dc05a20984998a96faf4fce9a"} Nov 02 18:02:18.841843 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.841791 5647 scope.go:122] "RemoveContainer" containerID="5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90" Nov 02 18:02:18.852577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.852558 5647 scope.go:122] "RemoveContainer" containerID="fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc" Nov 02 18:02:18.861716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.861693 5647 scope.go:122] "RemoveContainer" containerID="5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90" Nov 02 18:02:18.862037 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:02:18.862018 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90\": container with ID starting with 5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90 not found: ID does not exist" containerID="5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90" Nov 02 18:02:18.862082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.862047 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90"} err="failed to get container status \"5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90\": rpc error: code = NotFound desc = could not find container \"5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90\": container with ID starting with 5aad875b4fb15e4d053fcc3026ecbfa4723c711c8a0d9a56590c32d2bec16b90 not found: ID does not exist" Nov 02 18:02:18.862082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.862066 5647 scope.go:122] "RemoveContainer" containerID="fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc" Nov 02 18:02:18.862297 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:02:18.862280 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc\": container with ID starting with fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc not found: ID does not exist" containerID="fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc" Nov 02 18:02:18.862340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:18.862301 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc"} err="failed to get container status \"fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc\": rpc error: code = NotFound desc = could not find container \"fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc\": container with ID starting with fd8553adfbff5a05297ae230a1dc2a8cb59d665df3f188f864da6c7f974072dc not found: ID does not exist" Nov 02 18:02:19.015886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:19.015837 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9"] Nov 02 18:02:19.066708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:19.066666 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5a1e27-openshift-default-598d9f7-nbjj9"] Nov 02 18:02:19.082191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:19.082157 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" path="/var/lib/kubelet/pods/dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1/volumes" Nov 02 18:02:23.034353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.034304 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2"] Nov 02 18:02:23.034959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.034542 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" podUID="5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" containerName="maas-api" containerID="cri-o://0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52" gracePeriod=30 Nov 02 18:02:23.366013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.365988 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 18:02:23.498234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.498199 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk\" (UniqueName: \"kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk\") pod \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " Nov 02 18:02:23.498436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.498277 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs\") pod \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " Nov 02 18:02:23.498436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.498343 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca\") pod \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " Nov 02 18:02:23.498582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.498489 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\") pod \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\" (UID: \"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95\") " Nov 02 18:02:23.498766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.498718 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca" pod "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" (UID: "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:23.499148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.499120 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:23.503188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.500764 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls" pod "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" (UID: "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:23.503188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.501145 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs" pod "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" (UID: "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:02:23.503485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.503462 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk" pod "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" (UID: "5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95"). InnerVolumeSpecName "kube-api-access-48rlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:23.617446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.615341 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:23.617446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.615383 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rlk\" (UniqueName: \"kubernetes.io/projected/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-kube-api-access-48rlk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:23.617446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.617095 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:23.909170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.909079 5647 generic.go:356] "Generic (PLEG): container finished" podID="5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" containerID="0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52" exitCode=0 Nov 02 18:02:23.909170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.909136 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" Nov 02 18:02:23.909170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.909158 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" event={"ID":"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95","Type":"ContainerDied","Data":"0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52"} Nov 02 18:02:23.909436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.909216 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2" event={"ID":"5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95","Type":"ContainerDied","Data":"29e4a458f61ba01f6b01f83086d5a52778cae5367eb4f4e2b9ee403e9d1099eb"} Nov 02 18:02:23.909436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.909237 5647 scope.go:122] "RemoveContainer" containerID="0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52" Nov 02 18:02:23.920647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.920620 5647 scope.go:122] "RemoveContainer" containerID="0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52" Nov 02 18:02:23.920990 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:02:23.920968 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52\": container with ID starting with 0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52 not found: ID does not exist" containerID="0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52" Nov 02 18:02:23.921069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:23.921002 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52"} err="failed to get container status \"0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52\": rpc error: code = NotFound desc = could not find container \"0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52\": container with ID starting with 0445e0546c56f912e7cab5a0cd190a13171cbd48897da92afcb8a4e6f50a0b52 not found: ID does not exist" Nov 02 18:02:24.084678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:24.084595 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2"] Nov 02 18:02:24.132543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:24.132506 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f42381-84f79fc8f6-gsst2"] Nov 02 18:02:25.080962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:25.080929 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" path="/var/lib/kubelet/pods/5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95/volumes" Nov 02 18:02:27.776010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:27.775965 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt"] Nov 02 18:02:27.776438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:27.776209 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" containerID="cri-o://ecf9ce106d685fa08be405822040958bf55a4b4ddd9dafdb3ad8f311a614dca9" gracePeriod=30 Nov 02 18:02:30.817478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:30.817441 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:02:32.993134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:32.993100 5647 generic.go:356] "Generic (PLEG): container finished" podID="89fd7905-7063-46bf-9e1f-1d7755094293" containerID="ecf9ce106d685fa08be405822040958bf55a4b4ddd9dafdb3ad8f311a614dca9" exitCode=0 Nov 02 18:02:32.993568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:32.993170 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" event={"ID":"89fd7905-7063-46bf-9e1f-1d7755094293","Type":"ContainerDied","Data":"ecf9ce106d685fa08be405822040958bf55a4b4ddd9dafdb3ad8f311a614dca9"} Nov 02 18:02:32.993568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:32.993254 5647 scope.go:122] "RemoveContainer" containerID="938e9e2a92eb3e4029809a35c41949c4c9a9db74c63bdcc6cb30c9a1aa7d6092" Nov 02 18:02:33.038138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.038080 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 18:02:33.062979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.062949 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.062986 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063025 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063123 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063160 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063228 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063272 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063292 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:33.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063302 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:33.063608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063468 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:33.063608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063498 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063562 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs\") pod \"89fd7905-7063-46bf-9e1f-1d7755094293\" (UID: \"89fd7905-7063-46bf-9e1f-1d7755094293\") " Nov 02 18:02:33.063981 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063863 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:33.063981 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.063897 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:02:33.064358 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.064334 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.064437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.064363 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89fd7905-7063-46bf-9e1f-1d7755094293-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.064437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.064379 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.064437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.064394 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.064437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.064407 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.065534 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.065509 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:02:33.065650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.065548 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:33.065937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.065919 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:02:33.065937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.065924 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h" pod "89fd7905-7063-46bf-9e1f-1d7755094293" (UID: "89fd7905-7063-46bf-9e1f-1d7755094293"). InnerVolumeSpecName "kube-api-access-czv5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:02:33.165441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.165410 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89fd7905-7063-46bf-9e1f-1d7755094293-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.165441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.165439 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89fd7905-7063-46bf-9e1f-1d7755094293-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.165441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.165448 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czv5h\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-kube-api-access-czv5h\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:33.165694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:33.165459 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89fd7905-7063-46bf-9e1f-1d7755094293-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:02:34.000711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:34.000673 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" Nov 02 18:02:34.000711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:34.000694 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt" event={"ID":"89fd7905-7063-46bf-9e1f-1d7755094293","Type":"ContainerDied","Data":"ffc949ffd2db694a19498340a4fe45d75d20cd58259c41056b84c72e3941b239"} Nov 02 18:02:34.001441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:34.000807 5647 scope.go:122] "RemoveContainer" containerID="ecf9ce106d685fa08be405822040958bf55a4b4ddd9dafdb3ad8f311a614dca9" Nov 02 18:02:34.219419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:34.219385 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt"] Nov 02 18:02:34.222957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:34.222928 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f42381-openshift-default-65dd6989-95lrt"] Nov 02 18:02:35.081241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:35.081203 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" path="/var/lib/kubelet/pods/89fd7905-7063-46bf-9e1f-1d7755094293/volumes" Nov 02 18:02:35.947811 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:02:35.947775 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:02:56.793503 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.793464 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg"] Nov 02 18:02:56.794283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794260 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9899c816-f442-49df-b19e-dc9d419d1718" containerName="payload-processing" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794285 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9899c816-f442-49df-b19e-dc9d419d1718" containerName="payload-processing" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794299 5647 cpu_manager.go:414] "containerMap: removing container" podUID="da3fab43-cd7c-486d-bc78-6c1616a78a22" containerName="payload-pre-processing" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794307 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da3fab43-cd7c-486d-bc78-6c1616a78a22" containerName="payload-pre-processing" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794323 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794330 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794345 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" containerName="maas-api" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794354 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" containerName="maas-api" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794367 5647 cpu_manager.go:414] "containerMap: removing container" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" Nov 02 18:02:56.794391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794375 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794412 5647 cpu_manager.go:414] "containerMap: removing container" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794419 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794431 5647 cpu_manager.go:414] "containerMap: removing container" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794438 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794455 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794462 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794477 5647 cpu_manager.go:414] "containerMap: removing container" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794485 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794500 5647 cpu_manager.go:414] "containerMap: removing container" podUID="fb401a53-d003-4c5e-84bf-224f48596f31" containerName="payload-pre-processing" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794507 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb401a53-d003-4c5e-84bf-224f48596f31" containerName="payload-pre-processing" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794518 5647 cpu_manager.go:414] "containerMap: removing container" podUID="5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" containerName="maas-api" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794524 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" containerName="maas-api" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794671 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c989b4af-ae4f-4bc1-a8e2-8753a19cb76e" containerName="maas-api" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794684 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794698 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa9c3a4-2e34-4f5c-92c6-5fdb73937842" containerName="cleanup" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794709 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794716 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4cb11c-63ac-4f2f-9811-77f0b6eba6c6" containerName="cleanup" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794728 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb401a53-d003-4c5e-84bf-224f48596f31" containerName="payload-pre-processing" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794758 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdf24cc-ed90-4f7c-a8f1-6bb144d71d95" containerName="maas-api" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794771 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9899c816-f442-49df-b19e-dc9d419d1718" containerName="payload-processing" Nov 02 18:02:56.794906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.794781 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="da3fab43-cd7c-486d-bc78-6c1616a78a22" containerName="payload-pre-processing" Nov 02 18:02:56.798470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.798450 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:56.886239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.886197 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:56.886465 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.886377 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvdq\" (UniqueName: \"kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:56.902619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.902583 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f07b5b\"" Nov 02 18:02:56.902829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.902624 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f07b5b-dockercfg-pq5tf\"" Nov 02 18:02:56.906484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.906451 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg"] Nov 02 18:02:56.987136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.987102 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvdq\" (UniqueName: \"kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:56.987348 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.987264 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:56.987966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:56.987946 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:57.046669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:57.046598 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvdq\" (UniqueName: \"kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq\") pod \"payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:57.112332 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:57.112302 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:02:57.489144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:57.489107 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg"] Nov 02 18:02:57.491055 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:02:57.491024 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d63b5c_a9ab_4ffa_8515_10359bdea516.slice/crio-bc61587127d8cf86748c3f060a3146d46306eaf181d190b0ce5ad3eab85d7b7e WatchSource:0}: Error finding container bc61587127d8cf86748c3f060a3146d46306eaf181d190b0ce5ad3eab85d7b7e: Status 404 returned error can't find the container with id bc61587127d8cf86748c3f060a3146d46306eaf181d190b0ce5ad3eab85d7b7e Nov 02 18:02:58.164258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:58.164221 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" event={"ID":"c6d63b5c-a9ab-4ffa-8515-10359bdea516","Type":"ContainerStarted","Data":"67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203"} Nov 02 18:02:58.164791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:58.164284 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" event={"ID":"c6d63b5c-a9ab-4ffa-8515-10359bdea516","Type":"ContainerStarted","Data":"bc61587127d8cf86748c3f060a3146d46306eaf181d190b0ce5ad3eab85d7b7e"} Nov 02 18:02:58.164791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:02:58.164419 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:03:00.855691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:00.855631 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" podStartSLOduration=4.855615204 podStartE2EDuration="4.855615204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:58.293217917 +0000 UTC m=+2365.366745121" watchObservedRunningTime="2026-11-02 18:03:00.855615204 +0000 UTC m=+2367.929142814" Nov 02 18:03:00.856305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:00.856284 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:03:05.947929 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:05.947885 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:03:07.149537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.149502 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk"] Nov 02 18:03:07.149979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.149714 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" podUID="9b9c2867-5a09-4566-9dec-d0872e123f12" containerName="maas-api" containerID="cri-o://61da15bf67adcd77b47f0a1b7e27c3b0e50041fe59c120d88cb60b3c2d5c83f2" gracePeriod=30 Nov 02 18:03:07.278204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.278169 5647 generic.go:356] "Generic (PLEG): container finished" podID="9b9c2867-5a09-4566-9dec-d0872e123f12" containerID="61da15bf67adcd77b47f0a1b7e27c3b0e50041fe59c120d88cb60b3c2d5c83f2" exitCode=0 Nov 02 18:03:07.278399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.278248 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" event={"ID":"9b9c2867-5a09-4566-9dec-d0872e123f12","Type":"ContainerDied","Data":"61da15bf67adcd77b47f0a1b7e27c3b0e50041fe59c120d88cb60b3c2d5c83f2"} Nov 02 18:03:07.439636 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.439612 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 18:03:07.533263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.533226 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca\") pod \"9b9c2867-5a09-4566-9dec-d0872e123f12\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " Nov 02 18:03:07.533455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.533280 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs\") pod \"9b9c2867-5a09-4566-9dec-d0872e123f12\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " Nov 02 18:03:07.533505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.533451 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df\" (UniqueName: \"kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df\") pod \"9b9c2867-5a09-4566-9dec-d0872e123f12\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " Nov 02 18:03:07.533505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.533499 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls\") pod \"9b9c2867-5a09-4566-9dec-d0872e123f12\" (UID: \"9b9c2867-5a09-4566-9dec-d0872e123f12\") " Nov 02 18:03:07.533619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.533596 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca" pod "9b9c2867-5a09-4566-9dec-d0872e123f12" (UID: "9b9c2867-5a09-4566-9dec-d0872e123f12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:07.533978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.533958 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b9c2867-5a09-4566-9dec-d0872e123f12-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:07.535672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.535643 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs" pod "9b9c2867-5a09-4566-9dec-d0872e123f12" (UID: "9b9c2867-5a09-4566-9dec-d0872e123f12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:07.535816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.535697 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls" pod "9b9c2867-5a09-4566-9dec-d0872e123f12" (UID: "9b9c2867-5a09-4566-9dec-d0872e123f12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:07.535816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.535748 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df" pod "9b9c2867-5a09-4566-9dec-d0872e123f12" (UID: "9b9c2867-5a09-4566-9dec-d0872e123f12"). InnerVolumeSpecName "kube-api-access-zv9df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:07.635148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.635111 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv9df\" (UniqueName: \"kubernetes.io/projected/9b9c2867-5a09-4566-9dec-d0872e123f12-kube-api-access-zv9df\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:07.635148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.635147 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:07.635371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:07.635157 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b9c2867-5a09-4566-9dec-d0872e123f12-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:08.286349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:08.286305 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" event={"ID":"9b9c2867-5a09-4566-9dec-d0872e123f12","Type":"ContainerDied","Data":"cda2a69f11518542babe66be368d0bc40e48739060ac6af794273fa214dcff8a"} Nov 02 18:03:08.286349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:08.286338 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk" Nov 02 18:03:08.286950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:08.286384 5647 scope.go:122] "RemoveContainer" containerID="61da15bf67adcd77b47f0a1b7e27c3b0e50041fe59c120d88cb60b3c2d5c83f2" Nov 02 18:03:08.454677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:08.454635 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk"] Nov 02 18:03:08.506489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:08.506453 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ccf013-7f6f6bb5d9-6c5pk"] Nov 02 18:03:09.081878 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:09.081847 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9c2867-5a09-4566-9dec-d0872e123f12" path="/var/lib/kubelet/pods/9b9c2867-5a09-4566-9dec-d0872e123f12/volumes" Nov 02 18:03:09.173448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:09.173406 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:03:15.083533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.083501 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h"] Nov 02 18:03:15.084098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.084083 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9b9c2867-5a09-4566-9dec-d0872e123f12" containerName="maas-api" Nov 02 18:03:15.084163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.084100 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9c2867-5a09-4566-9dec-d0872e123f12" containerName="maas-api" Nov 02 18:03:15.084201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.084188 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8aa7ff-0db9-4cea-a1a6-0b24469e7ad1" containerName="istio-proxy" Nov 02 18:03:15.084239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.084202 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9c2867-5a09-4566-9dec-d0872e123f12" containerName="maas-api" Nov 02 18:03:15.084239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.084212 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fd7905-7063-46bf-9e1f-1d7755094293" containerName="istio-proxy" Nov 02 18:03:15.089058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.089036 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.142418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.142383 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-948451-openshift-default-dockercfg-cfwcz\"" Nov 02 18:03:15.196647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.196610 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h"] Nov 02 18:03:15.241823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.241716 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.242430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.242344 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.242850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.242804 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.243075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.243047 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.243279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.243249 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.243475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.243447 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.243676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.243652 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq7sd\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.243875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.243857 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.244155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.244124 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349667 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349614 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349679 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349713 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349758 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349797 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq7sd\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349832 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.349907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349886 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.350253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.349985 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.350253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.350044 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.350521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.350493 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.351255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.351226 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.351381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.351270 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.351542 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.351520 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.351956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.351931 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.355163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.354622 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.357993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.357927 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.414636 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.414597 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.414913 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.414891 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq7sd\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd\") pod \"e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:15.702230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:15.702141 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:16.176024 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:03:16.175983 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080fb4bf_8385_4e94_a757_adf3b771f4fa.slice/crio-7e2ed35737273fc444a4657412b86ce4fad8c9ce14a6d418d6bc9f2d37a4b9f0 WatchSource:0}: Error finding container 7e2ed35737273fc444a4657412b86ce4fad8c9ce14a6d418d6bc9f2d37a4b9f0: Status 404 returned error can't find the container with id 7e2ed35737273fc444a4657412b86ce4fad8c9ce14a6d418d6bc9f2d37a4b9f0 Nov 02 18:03:16.184331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.183134 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h"] Nov 02 18:03:16.186949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.186910 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:03:16.187161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.186984 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:03:16.187208 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.187184 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:03:16.365511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.365476 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" event={"ID":"080fb4bf-8385-4e94-a757-adf3b771f4fa","Type":"ContainerStarted","Data":"979deb33e252e5b39dd1304aab8416b050249f73521bb8c9590af67686213b43"} Nov 02 18:03:16.365702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.365540 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" event={"ID":"080fb4bf-8385-4e94-a757-adf3b771f4fa","Type":"ContainerStarted","Data":"7e2ed35737273fc444a4657412b86ce4fad8c9ce14a6d418d6bc9f2d37a4b9f0"} Nov 02 18:03:16.491808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.491690 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podStartSLOduration=1.4916749120000001 podStartE2EDuration="1.491674912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:16.489888668 +0000 UTC m=+2383.563415872" watchObservedRunningTime="2026-11-02 18:03:16.491674912 +0000 UTC m=+2383.565202462" Nov 02 18:03:16.703066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.703033 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:16.965143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.965106 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j"] Nov 02 18:03:16.969329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:16.969309 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.023184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.023147 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:03:17.023409 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.023358 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:03:17.025652 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.025628 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j"] Nov 02 18:03:17.071128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.071094 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4n9\" (UniqueName: \"kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.071312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.071178 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.071312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.071282 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.173172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.173134 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4n9\" (UniqueName: \"kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.173393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.173230 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.173393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.173275 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.174106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.174076 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.175761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.175720 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.232001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.231921 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4n9\" (UniqueName: \"kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9\") pod \"maas-api-revoke-sub-7811dfc0-9j77j\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.283635 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.283593 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:17.664568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.664542 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j"] Nov 02 18:03:17.666384 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:03:17.666355 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc759a51_cbd6_4917_b050_806dc5481899.slice/crio-eb4e0b0de0120ebc639ad36362240f3cb2eb92d9b33eb9d72e8b0b37c8809a5f WatchSource:0}: Error finding container eb4e0b0de0120ebc639ad36362240f3cb2eb92d9b33eb9d72e8b0b37c8809a5f: Status 404 returned error can't find the container with id eb4e0b0de0120ebc639ad36362240f3cb2eb92d9b33eb9d72e8b0b37c8809a5f Nov 02 18:03:17.708900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:17.708874 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:18.381146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:18.381106 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" event={"ID":"cc759a51-cbd6-4917-b050-806dc5481899","Type":"ContainerStarted","Data":"228ffde832da1e310208a6c253f9d282d6e877548848b21f692c0f561c8b713e"} Nov 02 18:03:18.381668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:18.381173 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" event={"ID":"cc759a51-cbd6-4917-b050-806dc5481899","Type":"ContainerStarted","Data":"eb4e0b0de0120ebc639ad36362240f3cb2eb92d9b33eb9d72e8b0b37c8809a5f"} Nov 02 18:03:18.381668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:18.381499 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:18.382495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:18.382475 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:03:18.496958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:18.496892 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" podStartSLOduration=2.496873776 podStartE2EDuration="2.496873776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:18.496405926 +0000 UTC m=+2385.569933130" watchObservedRunningTime="2026-11-02 18:03:18.496873776 +0000 UTC m=+2385.570400980" Nov 02 18:03:19.432080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:19.432032 5647 generic.go:356] "Generic (PLEG): container finished" podID="cc759a51-cbd6-4917-b050-806dc5481899" containerID="228ffde832da1e310208a6c253f9d282d6e877548848b21f692c0f561c8b713e" exitCode=0 Nov 02 18:03:19.433588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:19.433543 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" event={"ID":"cc759a51-cbd6-4917-b050-806dc5481899","Type":"ContainerDied","Data":"228ffde832da1e310208a6c253f9d282d6e877548848b21f692c0f561c8b713e"} Nov 02 18:03:20.593453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.593419 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:20.739373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.739277 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca\") pod \"cc759a51-cbd6-4917-b050-806dc5481899\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " Nov 02 18:03:20.739569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.739417 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca\") pod \"cc759a51-cbd6-4917-b050-806dc5481899\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " Nov 02 18:03:20.739569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.739564 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9\" (UniqueName: \"kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9\") pod \"cc759a51-cbd6-4917-b050-806dc5481899\" (UID: \"cc759a51-cbd6-4917-b050-806dc5481899\") " Nov 02 18:03:20.739908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.739807 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca" pod "cc759a51-cbd6-4917-b050-806dc5481899" (UID: "cc759a51-cbd6-4917-b050-806dc5481899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:20.740089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.740068 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:20.742117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.742090 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9" pod "cc759a51-cbd6-4917-b050-806dc5481899" (UID: "cc759a51-cbd6-4917-b050-806dc5481899"). InnerVolumeSpecName "kube-api-access-7w4n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:20.745242 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.745216 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca" pod "cc759a51-cbd6-4917-b050-806dc5481899" (UID: "cc759a51-cbd6-4917-b050-806dc5481899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:03:20.841107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.841070 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4n9\" (UniqueName: \"kubernetes.io/projected/cc759a51-cbd6-4917-b050-806dc5481899-kube-api-access-7w4n9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:20.841107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:20.841102 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc759a51-cbd6-4917-b050-806dc5481899-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:21.446933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:21.446841 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" event={"ID":"cc759a51-cbd6-4917-b050-806dc5481899","Type":"ContainerDied","Data":"eb4e0b0de0120ebc639ad36362240f3cb2eb92d9b33eb9d72e8b0b37c8809a5f"} Nov 02 18:03:21.446933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:21.446878 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j" Nov 02 18:03:21.446933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:21.446888 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb4e0b0de0120ebc639ad36362240f3cb2eb92d9b33eb9d72e8b0b37c8809a5f" Nov 02 18:03:23.054690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.054658 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs"] Nov 02 18:03:23.055242 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.055228 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cc759a51-cbd6-4917-b050-806dc5481899" containerName="revoke-keys" Nov 02 18:03:23.055284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.055243 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc759a51-cbd6-4917-b050-806dc5481899" containerName="revoke-keys" Nov 02 18:03:23.055346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.055338 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc759a51-cbd6-4917-b050-806dc5481899" containerName="revoke-keys" Nov 02 18:03:23.058704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.058686 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.080245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.080211 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.080410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.080344 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7wrg\" (UniqueName: \"kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg\") pod \"payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.115143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.115112 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs"] Nov 02 18:03:23.169491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.169437 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-948451-dockercfg-75dfw\"" Nov 02 18:03:23.169925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.169905 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-948451\"" Nov 02 18:03:23.222000 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.220953 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.222632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.222598 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7wrg\" (UniqueName: \"kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg\") pod \"payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.225569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.225539 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.235372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.235330 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk"] Nov 02 18:03:23.240567 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.240544 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.261006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.260975 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk"] Nov 02 18:03:23.283335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.283308 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7wrg\" (UniqueName: \"kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg\") pod \"payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.324318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.324238 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.324467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.324412 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chrc\" (UniqueName: \"kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc\") pod \"payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.372607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.372573 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:23.425690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.425635 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.426420 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.425787 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chrc\" (UniqueName: \"kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc\") pod \"payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.426681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.426658 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.484811 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.484781 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chrc\" (UniqueName: \"kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc\") pod \"payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.554462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.554423 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:23.775850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.775778 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs"] Nov 02 18:03:23.805762 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:03:23.800807 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e7997f_5dc9_42f3_bedc_5cb477431614.slice/crio-a076f3af6443f73eda6311a80c64e43b81298b259f14735af2474a225104cf49 WatchSource:0}: Error finding container a076f3af6443f73eda6311a80c64e43b81298b259f14735af2474a225104cf49: Status 404 returned error can't find the container with id a076f3af6443f73eda6311a80c64e43b81298b259f14735af2474a225104cf49 Nov 02 18:03:23.937172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:23.937148 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk"] Nov 02 18:03:23.938689 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:03:23.938665 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887fa935_2841_49f7_ad11_c08efd7283a8.slice/crio-fc3c7d3298299715670d0458fec349251ad66009ee237f8df90e0034912a19eb WatchSource:0}: Error finding container fc3c7d3298299715670d0458fec349251ad66009ee237f8df90e0034912a19eb: Status 404 returned error can't find the container with id fc3c7d3298299715670d0458fec349251ad66009ee237f8df90e0034912a19eb Nov 02 18:03:24.467631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.467594 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" event={"ID":"15e7997f-5dc9-42f3-bedc-5cb477431614","Type":"ContainerStarted","Data":"bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561"} Nov 02 18:03:24.468153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.467666 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:24.468153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.467689 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" event={"ID":"15e7997f-5dc9-42f3-bedc-5cb477431614","Type":"ContainerStarted","Data":"a076f3af6443f73eda6311a80c64e43b81298b259f14735af2474a225104cf49"} Nov 02 18:03:24.469500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.469473 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" event={"ID":"887fa935-2841-49f7-ad11-c08efd7283a8","Type":"ContainerStarted","Data":"250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429"} Nov 02 18:03:24.469650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.469518 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" event={"ID":"887fa935-2841-49f7-ad11-c08efd7283a8","Type":"ContainerStarted","Data":"fc3c7d3298299715670d0458fec349251ad66009ee237f8df90e0034912a19eb"} Nov 02 18:03:24.469650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.469572 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:24.585850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.585794 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" podStartSLOduration=1.585779944 podStartE2EDuration="1.585779944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:24.58511811 +0000 UTC m=+2391.658645314" watchObservedRunningTime="2026-11-02 18:03:24.585779944 +0000 UTC m=+2391.659307146" Nov 02 18:03:24.725788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:24.725615 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" podStartSLOduration=1.725595474 podStartE2EDuration="1.725595474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:03:24.723294593 +0000 UTC m=+2391.796821796" watchObservedRunningTime="2026-11-02 18:03:24.725595474 +0000 UTC m=+2391.799122678" Nov 02 18:03:31.993902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:31.993863 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw"] Nov 02 18:03:31.994362 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:31.994121 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" containerID="cri-o://d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180" gracePeriod=30 Nov 02 18:03:33.097112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:33.097076 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7"] Nov 02 18:03:33.097546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:33.097318 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" containerID="cri-o://d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b" gracePeriod=30 Nov 02 18:03:34.250711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:34.250684 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:03:34.252546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:34.252525 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:03:35.478434 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:35.478408 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:03:35.478951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:35.478474 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:03:35.953493 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:35.953451 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:03:37.260960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.260825 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 18:03:37.304292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.304260 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc"] Nov 02 18:03:37.351221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351184 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351243 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351294 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351322 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351364 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351447 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351562 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351602 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.351870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351557 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:37.351870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351591 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:37.351870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351685 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:37.351870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351802 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:37.351870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.351763 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token\") pod \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\" (UID: \"b8a6c65e-5c29-4325-8cc6-1ecef065dba5\") " Nov 02 18:03:37.352490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.352449 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:37.352490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.352484 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.352662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.352506 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.352662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.352521 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.352662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.352534 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.354373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.354351 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:37.354486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.354391 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:03:37.354486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.354477 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "kube-api-access-2bh9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:37.354695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.354676 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token" pod "b8a6c65e-5c29-4325-8cc6-1ecef065dba5" (UID: "b8a6c65e-5c29-4325-8cc6-1ecef065dba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:37.416081 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.416040 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6"] Nov 02 18:03:37.453837 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.453804 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.453837 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.453834 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.454061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.453846 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.454061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.453862 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bh9c\" (UniqueName: \"kubernetes.io/projected/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-kube-api-access-2bh9c\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.454061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.453876 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a6c65e-5c29-4325-8cc6-1ecef065dba5-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:37.575273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.575179 5647 generic.go:356] "Generic (PLEG): container finished" podID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerID="d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180" exitCode=0 Nov 02 18:03:37.575273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.575236 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" Nov 02 18:03:37.575273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.575252 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" event={"ID":"b8a6c65e-5c29-4325-8cc6-1ecef065dba5","Type":"ContainerDied","Data":"d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180"} Nov 02 18:03:37.575584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.575322 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw" event={"ID":"b8a6c65e-5c29-4325-8cc6-1ecef065dba5","Type":"ContainerDied","Data":"57ccbc899555f6593e416f09050758dc950db24399232bf2ae155e29cc27a904"} Nov 02 18:03:37.575584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.575351 5647 scope.go:122] "RemoveContainer" containerID="d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180" Nov 02 18:03:37.588196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.588177 5647 scope.go:122] "RemoveContainer" containerID="89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b" Nov 02 18:03:37.597794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.597770 5647 scope.go:122] "RemoveContainer" containerID="d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180" Nov 02 18:03:37.598104 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:37.598082 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180\": container with ID starting with d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180 not found: ID does not exist" containerID="d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180" Nov 02 18:03:37.598178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.598112 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180"} err="failed to get container status \"d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180\": rpc error: code = NotFound desc = could not find container \"d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180\": container with ID starting with d979bb3ee893d89534e2842a995d2a24523823766226d9c6f74e68f618a3a180 not found: ID does not exist" Nov 02 18:03:37.598178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.598132 5647 scope.go:122] "RemoveContainer" containerID="89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b" Nov 02 18:03:37.598391 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:37.598369 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b\": container with ID starting with 89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b not found: ID does not exist" containerID="89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b" Nov 02 18:03:37.598459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.598399 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b"} err="failed to get container status \"89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b\": rpc error: code = NotFound desc = could not find container \"89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b\": container with ID starting with 89cb89dee22bcb426a5e825a98efb27de655e607f0dc48a8435104521be67d2b not found: ID does not exist" Nov 02 18:03:37.779913 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.779879 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw"] Nov 02 18:03:37.785804 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:37.785775 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-ffmbw"] Nov 02 18:03:38.368870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.368803 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 18:03:38.466453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466419 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466465 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466552 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466661 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466770 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:38.466834 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466787 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466890 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466874 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466924 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.466992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466931 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:38.466992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466965 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:38.467111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.466992 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.467111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467034 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data\") pod \"653639db-490a-41c7-9ef1-a067d45c5f7b\" (UID: \"653639db-490a-41c7-9ef1-a067d45c5f7b\") " Nov 02 18:03:38.467244 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467223 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:38.467353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467331 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:38.467923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467901 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.468006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467928 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/653639db-490a-41c7-9ef1-a067d45c5f7b-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.468006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467947 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.468006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467966 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.468006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.467982 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.469118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.469097 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:03:38.469311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.469292 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "kube-api-access-2zcjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:38.469464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.469439 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:38.469464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.469440 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo" pod "653639db-490a-41c7-9ef1-a067d45c5f7b" (UID: "653639db-490a-41c7-9ef1-a067d45c5f7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:03:38.568687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.568651 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.568687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.568680 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.568687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.568690 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/653639db-490a-41c7-9ef1-a067d45c5f7b-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.568962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.568699 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zcjc\" (UniqueName: \"kubernetes.io/projected/653639db-490a-41c7-9ef1-a067d45c5f7b-kube-api-access-2zcjc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:38.585036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.584985 5647 generic.go:356] "Generic (PLEG): container finished" podID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerID="d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b" exitCode=0 Nov 02 18:03:38.585192 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.585116 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" event={"ID":"653639db-490a-41c7-9ef1-a067d45c5f7b","Type":"ContainerDied","Data":"d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b"} Nov 02 18:03:38.585274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.585225 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" event={"ID":"653639db-490a-41c7-9ef1-a067d45c5f7b","Type":"ContainerDied","Data":"8c9a7fff6d591797296165e84d831edb35330fcf4129928427b56de26b92e010"} Nov 02 18:03:38.585331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.585267 5647 scope.go:122] "RemoveContainer" containerID="d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b" Nov 02 18:03:38.585595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.585469 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7" Nov 02 18:03:38.600176 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.600152 5647 scope.go:122] "RemoveContainer" containerID="31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f" Nov 02 18:03:38.610876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.610853 5647 scope.go:122] "RemoveContainer" containerID="d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b" Nov 02 18:03:38.611200 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:38.611177 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b\": container with ID starting with d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b not found: ID does not exist" containerID="d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b" Nov 02 18:03:38.611294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.611212 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b"} err="failed to get container status \"d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b\": rpc error: code = NotFound desc = could not find container \"d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b\": container with ID starting with d84566f157db398c14119c36a43beced39aa686af7fbea99a25b9b77e87be05b not found: ID does not exist" Nov 02 18:03:38.611294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.611238 5647 scope.go:122] "RemoveContainer" containerID="31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f" Nov 02 18:03:38.611517 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:38.611499 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f\": container with ID starting with 31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f not found: ID does not exist" containerID="31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f" Nov 02 18:03:38.611555 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.611525 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f"} err="failed to get container status \"31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f\": rpc error: code = NotFound desc = could not find container \"31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f\": container with ID starting with 31de5d2ef5756558efecd3cfa543625899f119adcf7c6a842a4e8c41e2e06d3f not found: ID does not exist" Nov 02 18:03:38.772330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.772217 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7"] Nov 02 18:03:38.774709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:38.774680 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d5924c-openshift-default-86bcffb5c5-kkpd7"] Nov 02 18:03:39.083289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:39.083200 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" path="/var/lib/kubelet/pods/653639db-490a-41c7-9ef1-a067d45c5f7b/volumes" Nov 02 18:03:39.083699 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:39.083683 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" path="/var/lib/kubelet/pods/b8a6c65e-5c29-4325-8cc6-1ecef065dba5/volumes" Nov 02 18:03:42.305423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.305384 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" podUID="6e43c32f-b34b-4059-841a-f0f7eb578eb5" containerName="payload-processing" containerID="cri-o://1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862" gracePeriod=25 Nov 02 18:03:42.416984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.416939 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" podUID="a84587b3-79b0-441d-b159-59411f0d343f" containerName="payload-pre-processing" containerID="cri-o://9c1e13c955ebb9cc1d067dc6d965abca3c4990af26306c119d18c06b98db6ff0" gracePeriod=25 Nov 02 18:03:42.582267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.582235 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 18:03:42.617095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.617061 5647 generic.go:356] "Generic (PLEG): container finished" podID="a84587b3-79b0-441d-b159-59411f0d343f" containerID="9c1e13c955ebb9cc1d067dc6d965abca3c4990af26306c119d18c06b98db6ff0" exitCode=0 Nov 02 18:03:42.617308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.617131 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" event={"ID":"a84587b3-79b0-441d-b159-59411f0d343f","Type":"ContainerDied","Data":"9c1e13c955ebb9cc1d067dc6d965abca3c4990af26306c119d18c06b98db6ff0"} Nov 02 18:03:42.618751 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.618710 5647 generic.go:356] "Generic (PLEG): container finished" podID="6e43c32f-b34b-4059-841a-f0f7eb578eb5" containerID="1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862" exitCode=0 Nov 02 18:03:42.618897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.618780 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" Nov 02 18:03:42.618897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.618779 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" event={"ID":"6e43c32f-b34b-4059-841a-f0f7eb578eb5","Type":"ContainerDied","Data":"1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862"} Nov 02 18:03:42.618897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.618854 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc" event={"ID":"6e43c32f-b34b-4059-841a-f0f7eb578eb5","Type":"ContainerDied","Data":"af30cedbe41bc6799676df28c3354037557cdf21cd7b2722273e4ddaf616bb7c"} Nov 02 18:03:42.618897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.618886 5647 scope.go:122] "RemoveContainer" containerID="1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862" Nov 02 18:03:42.631170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.631036 5647 scope.go:122] "RemoveContainer" containerID="1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862" Nov 02 18:03:42.631452 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:03:42.631420 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862\": container with ID starting with 1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862 not found: ID does not exist" containerID="1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862" Nov 02 18:03:42.631563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.631467 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862"} err="failed to get container status \"1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862\": rpc error: code = NotFound desc = could not find container \"1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862\": container with ID starting with 1343fee7ed50907d6c0ad9d73719a45c8b4c138beab7e7aef1e25ae9c4f3e862 not found: ID does not exist" Nov 02 18:03:42.682137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.682109 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 18:03:42.717215 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.717185 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26\" (UniqueName: \"kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26\") pod \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " Nov 02 18:03:42.717418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.717229 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume\") pod \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\" (UID: \"6e43c32f-b34b-4059-841a-f0f7eb578eb5\") " Nov 02 18:03:42.717793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.717721 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume" pod "6e43c32f-b34b-4059-841a-f0f7eb578eb5" (UID: "6e43c32f-b34b-4059-841a-f0f7eb578eb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:42.719499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.719477 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26" pod "6e43c32f-b34b-4059-841a-f0f7eb578eb5" (UID: "6e43c32f-b34b-4059-841a-f0f7eb578eb5"). InnerVolumeSpecName "kube-api-access-w6c26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:42.818801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.818707 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume\") pod \"a84587b3-79b0-441d-b159-59411f0d343f\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " Nov 02 18:03:42.818994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.818847 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6\" (UniqueName: \"kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6\") pod \"a84587b3-79b0-441d-b159-59411f0d343f\" (UID: \"a84587b3-79b0-441d-b159-59411f0d343f\") " Nov 02 18:03:42.819066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.819044 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume" pod "a84587b3-79b0-441d-b159-59411f0d343f" (UID: "a84587b3-79b0-441d-b159-59411f0d343f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:03:42.819306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.819286 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a84587b3-79b0-441d-b159-59411f0d343f-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:42.819381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.819311 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6c26\" (UniqueName: \"kubernetes.io/projected/6e43c32f-b34b-4059-841a-f0f7eb578eb5-kube-api-access-w6c26\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:42.819381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.819326 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e43c32f-b34b-4059-841a-f0f7eb578eb5-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:42.821035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.821017 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6" pod "a84587b3-79b0-441d-b159-59411f0d343f" (UID: "a84587b3-79b0-441d-b159-59411f0d343f"). InnerVolumeSpecName "kube-api-access-p7gs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:03:42.920225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:42.920186 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7gs6\" (UniqueName: \"kubernetes.io/projected/a84587b3-79b0-441d-b159-59411f0d343f-kube-api-access-p7gs6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:03:43.040752 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.040707 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc"] Nov 02 18:03:43.048464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.048429 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-fd04fa-f57dd4d65-8s6jc"] Nov 02 18:03:43.081758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.081669 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e43c32f-b34b-4059-841a-f0f7eb578eb5" path="/var/lib/kubelet/pods/6e43c32f-b34b-4059-841a-f0f7eb578eb5/volumes" Nov 02 18:03:43.626237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.626206 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" Nov 02 18:03:43.626237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.626213 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6" event={"ID":"a84587b3-79b0-441d-b159-59411f0d343f","Type":"ContainerDied","Data":"7ab9c8b0c55a3e34f917dee4ac5c5d84c90ff7551fb2cad5e7122ad47339a391"} Nov 02 18:03:43.626857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.626274 5647 scope.go:122] "RemoveContainer" containerID="9c1e13c955ebb9cc1d067dc6d965abca3c4990af26306c119d18c06b98db6ff0" Nov 02 18:03:43.793417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.793382 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6"] Nov 02 18:03:43.844120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:43.844087 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-fd04fa-6d7fd96f79-qf2j6"] Nov 02 18:03:45.064318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:45.064287 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:03:45.081281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:45.081246 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84587b3-79b0-441d-b159-59411f0d343f" path="/var/lib/kubelet/pods/a84587b3-79b0-441d-b159-59411f0d343f/volumes" Nov 02 18:03:49.343078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:49.343037 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9"] Nov 02 18:03:49.453698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:49.453630 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l"] Nov 02 18:03:54.343916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:54.343849 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerName="payload-processing" containerID="cri-o://9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a" gracePeriod=25 Nov 02 18:03:54.457839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:03:54.457779 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerName="payload-pre-processing" containerID="cri-o://f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826" gracePeriod=25 Nov 02 18:04:00.425120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:00.425079 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:04:00.425616 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:00.425090 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:04:05.947595 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:05.947555 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:04:06.986893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:06.986857 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d"] Nov 02 18:04:06.987334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:06.987066 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" podUID="79b91a83-b6fa-4680-b84f-11366154d5a6" containerName="maas-api" containerID="cri-o://ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a" gracePeriod=30 Nov 02 18:04:07.263047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.263020 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 18:04:07.321778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.321715 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp\" (UniqueName: \"kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp\") pod \"79b91a83-b6fa-4680-b84f-11366154d5a6\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " Nov 02 18:04:07.321947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.321792 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\") pod \"79b91a83-b6fa-4680-b84f-11366154d5a6\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " Nov 02 18:04:07.322026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.322001 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca\") pod \"79b91a83-b6fa-4680-b84f-11366154d5a6\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " Nov 02 18:04:07.322093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.322051 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs\") pod \"79b91a83-b6fa-4680-b84f-11366154d5a6\" (UID: \"79b91a83-b6fa-4680-b84f-11366154d5a6\") " Nov 02 18:04:07.322452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.322426 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca" pod "79b91a83-b6fa-4680-b84f-11366154d5a6" (UID: "79b91a83-b6fa-4680-b84f-11366154d5a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:07.324240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.324211 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls" pod "79b91a83-b6fa-4680-b84f-11366154d5a6" (UID: "79b91a83-b6fa-4680-b84f-11366154d5a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:07.324352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.324210 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp" pod "79b91a83-b6fa-4680-b84f-11366154d5a6" (UID: "79b91a83-b6fa-4680-b84f-11366154d5a6"). InnerVolumeSpecName "kube-api-access-hvslp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:07.324460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.324441 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs" pod "79b91a83-b6fa-4680-b84f-11366154d5a6" (UID: "79b91a83-b6fa-4680-b84f-11366154d5a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:07.423492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.423451 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79b91a83-b6fa-4680-b84f-11366154d5a6-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:07.423492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.423487 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:07.423492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.423496 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvslp\" (UniqueName: \"kubernetes.io/projected/79b91a83-b6fa-4680-b84f-11366154d5a6-kube-api-access-hvslp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:07.423761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.423506 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79b91a83-b6fa-4680-b84f-11366154d5a6-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:07.788137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.788096 5647 generic.go:356] "Generic (PLEG): container finished" podID="79b91a83-b6fa-4680-b84f-11366154d5a6" containerID="ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a" exitCode=0 Nov 02 18:04:07.788328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.788147 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" Nov 02 18:04:07.788328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.788178 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" event={"ID":"79b91a83-b6fa-4680-b84f-11366154d5a6","Type":"ContainerDied","Data":"ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a"} Nov 02 18:04:07.788328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.788233 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d" event={"ID":"79b91a83-b6fa-4680-b84f-11366154d5a6","Type":"ContainerDied","Data":"cac51645ae88ce69f89eff99ca1df41214b5c8516de141db5e5e556d97d456e6"} Nov 02 18:04:07.788328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.788254 5647 scope.go:122] "RemoveContainer" containerID="ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a" Nov 02 18:04:07.799713 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.799688 5647 scope.go:122] "RemoveContainer" containerID="ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a" Nov 02 18:04:07.800040 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:07.800017 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a\": container with ID starting with ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a not found: ID does not exist" containerID="ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a" Nov 02 18:04:07.800098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.800050 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a"} err="failed to get container status \"ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a\": rpc error: code = NotFound desc = could not find container \"ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a\": container with ID starting with ddbafac0cadbb8ae5eeb7ddf4473cf8cdf43e1b7bcb70d9bdd41cc792a1b6f1a not found: ID does not exist" Nov 02 18:04:07.963250 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.963210 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d"] Nov 02 18:04:07.965585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:07.965557 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-fd04fa-544966f8db-2kr5d"] Nov 02 18:04:09.081292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:09.081262 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b91a83-b6fa-4680-b84f-11366154d5a6" path="/var/lib/kubelet/pods/79b91a83-b6fa-4680-b84f-11366154d5a6/volumes" Nov 02 18:04:10.424922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:10.424878 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:04:10.424922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:10.424910 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:04:11.620160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.620050 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p"] Nov 02 18:04:11.621006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.620979 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" Nov 02 18:04:11.621006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621006 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621041 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6e43c32f-b34b-4059-841a-f0f7eb578eb5" containerName="payload-processing" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621049 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e43c32f-b34b-4059-841a-f0f7eb578eb5" containerName="payload-processing" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621076 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621083 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621096 5647 cpu_manager.go:414] "containerMap: removing container" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621103 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621112 5647 cpu_manager.go:414] "containerMap: removing container" podUID="79b91a83-b6fa-4680-b84f-11366154d5a6" containerName="maas-api" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621119 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b91a83-b6fa-4680-b84f-11366154d5a6" containerName="maas-api" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621148 5647 cpu_manager.go:414] "containerMap: removing container" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621156 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621171 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a84587b3-79b0-441d-b159-59411f0d343f" containerName="payload-pre-processing" Nov 02 18:04:11.621205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621178 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84587b3-79b0-441d-b159-59411f0d343f" containerName="payload-pre-processing" Nov 02 18:04:11.621917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621356 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" Nov 02 18:04:11.621917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621371 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e43c32f-b34b-4059-841a-f0f7eb578eb5" containerName="payload-processing" Nov 02 18:04:11.621917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621385 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a6c65e-5c29-4325-8cc6-1ecef065dba5" containerName="istio-proxy" Nov 02 18:04:11.621917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621394 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" Nov 02 18:04:11.621917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621401 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b91a83-b6fa-4680-b84f-11366154d5a6" containerName="maas-api" Nov 02 18:04:11.621917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.621414 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84587b3-79b0-441d-b159-59411f0d343f" containerName="payload-pre-processing" Nov 02 18:04:11.626984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.626960 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.680820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.680787 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p"] Nov 02 18:04:11.682813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.682788 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.682969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.682831 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57rb5\" (UniqueName: \"kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.683035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.683001 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.728341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.728312 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:11.728492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.728368 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:04:11.784995 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.784957 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.785205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.785020 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57rb5\" (UniqueName: \"kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.785205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.785103 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.785901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.785876 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.787513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.787489 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.844363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.844323 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57rb5\" (UniqueName: \"kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5\") pod \"maas-api-revoke-sub-96c1ea65-x8h9p\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:11.943879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:11.943765 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:12.320844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:12.320808 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p"] Nov 02 18:04:12.321639 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:04:12.321617 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6211150_1a7a_4f0e_a5bd_0172dea625b1.slice/crio-2c977b15f35ea15bb44f7a13c8a0c8fff1d5c8783ed124ac416f38703131c7d7 WatchSource:0}: Error finding container 2c977b15f35ea15bb44f7a13c8a0c8fff1d5c8783ed124ac416f38703131c7d7: Status 404 returned error can't find the container with id 2c977b15f35ea15bb44f7a13c8a0c8fff1d5c8783ed124ac416f38703131c7d7 Nov 02 18:04:12.820536 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:12.820498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" event={"ID":"f6211150-1a7a-4f0e-a5bd-0172dea625b1","Type":"ContainerStarted","Data":"c60eb88b9a855679b418aa18db7e5d533e12779ffafefa8275a06270ea985aff"} Nov 02 18:04:12.820933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:12.820558 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" event={"ID":"f6211150-1a7a-4f0e-a5bd-0172dea625b1","Type":"ContainerStarted","Data":"2c977b15f35ea15bb44f7a13c8a0c8fff1d5c8783ed124ac416f38703131c7d7"} Nov 02 18:04:12.938177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:12.938072 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" podStartSLOduration=1.9380583260000002 podStartE2EDuration="1.938058326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:12.936607571 +0000 UTC m=+2440.010134777" watchObservedRunningTime="2026-11-02 18:04:12.938058326 +0000 UTC m=+2440.011585528" Nov 02 18:04:13.303200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.303160 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw"] Nov 02 18:04:13.303884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.303865 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="653639db-490a-41c7-9ef1-a067d45c5f7b" containerName="istio-proxy" Nov 02 18:04:13.307294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.307271 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.362880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.362846 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-948451-serving-cert\"" Nov 02 18:04:13.365717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.365681 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw"] Nov 02 18:04:13.404351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.404315 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.404576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.404386 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.404576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.404457 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmj8\" (UniqueName: \"kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.404708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.404601 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.506186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.506090 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.506186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.506146 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.506186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.506164 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmj8\" (UniqueName: \"kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.506432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.506223 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.506432 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:13.506241 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-serving-cert: secret "maas-api-e2e-ipp-b-948451-serving-cert" not found Nov 02 18:04:13.506432 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:13.506301 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls podName:21237a61-6e71-48d7-b08e-dc6e4b3ebeab nodeName:}" failed. No retries permitted until 2026-11-02 18:04:14.006284788 +0000 UTC m=+2441.079811972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls") pod "maas-api-e2e-ipp-b-948451-df7574b69-wknqw" (UID: "21237a61-6e71-48d7-b08e-dc6e4b3ebeab") : secret "maas-api-e2e-ipp-b-948451-serving-cert" not found Nov 02 18:04:13.506953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.506930 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.508897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.508875 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.567320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.567285 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmj8\" (UniqueName: \"kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:13.829015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.828929 5647 generic.go:356] "Generic (PLEG): container finished" podID="f6211150-1a7a-4f0e-a5bd-0172dea625b1" containerID="c60eb88b9a855679b418aa18db7e5d533e12779ffafefa8275a06270ea985aff" exitCode=0 Nov 02 18:04:13.829015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:13.828980 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" event={"ID":"f6211150-1a7a-4f0e-a5bd-0172dea625b1","Type":"ContainerDied","Data":"c60eb88b9a855679b418aa18db7e5d533e12779ffafefa8275a06270ea985aff"} Nov 02 18:04:14.019336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.019301 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:14.021918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.021895 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-948451-df7574b69-wknqw\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:14.221080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.221018 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:14.601026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.600990 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw"] Nov 02 18:04:14.608240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.608214 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:04:14.608893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.608811 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 18:04:14.726907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.726817 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69\" (UniqueName: \"kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69\") pod \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " Nov 02 18:04:14.727073 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.726915 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume\") pod \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\" (UID: \"66ba1257-b531-4d87-b7f8-e2982a7b24e6\") " Nov 02 18:04:14.728283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.728247 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume" pod "66ba1257-b531-4d87-b7f8-e2982a7b24e6" (UID: "66ba1257-b531-4d87-b7f8-e2982a7b24e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:14.728849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.728442 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 18:04:14.730766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.730702 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69" pod "66ba1257-b531-4d87-b7f8-e2982a7b24e6" (UID: "66ba1257-b531-4d87-b7f8-e2982a7b24e6"). InnerVolumeSpecName "kube-api-access-vrt69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:14.828120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.828081 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume\") pod \"78c37530-b9f9-432c-8115-1ab575ef0eef\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " Nov 02 18:04:14.828298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.828132 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf\" (UniqueName: \"kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf\") pod \"78c37530-b9f9-432c-8115-1ab575ef0eef\" (UID: \"78c37530-b9f9-432c-8115-1ab575ef0eef\") " Nov 02 18:04:14.828415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.828403 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt69\" (UniqueName: \"kubernetes.io/projected/66ba1257-b531-4d87-b7f8-e2982a7b24e6-kube-api-access-vrt69\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:14.828460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.828417 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66ba1257-b531-4d87-b7f8-e2982a7b24e6-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:14.828493 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.828441 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume" pod "78c37530-b9f9-432c-8115-1ab575ef0eef" (UID: "78c37530-b9f9-432c-8115-1ab575ef0eef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:14.830497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.830469 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf" pod "78c37530-b9f9-432c-8115-1ab575ef0eef" (UID: "78c37530-b9f9-432c-8115-1ab575ef0eef"). InnerVolumeSpecName "kube-api-access-jcxkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:14.836357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.836328 5647 generic.go:356] "Generic (PLEG): container finished" podID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerID="9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a" exitCode=0 Nov 02 18:04:14.836486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.836377 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" Nov 02 18:04:14.836486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.836414 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" event={"ID":"66ba1257-b531-4d87-b7f8-e2982a7b24e6","Type":"ContainerDied","Data":"9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a"} Nov 02 18:04:14.836486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.836477 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9" event={"ID":"66ba1257-b531-4d87-b7f8-e2982a7b24e6","Type":"ContainerDied","Data":"066da305b5e16dd091599cc2abd5ffc3bd447e6ce483afbdae29be82044cb04b"} Nov 02 18:04:14.836680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.836507 5647 scope.go:122] "RemoveContainer" containerID="9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a" Nov 02 18:04:14.837799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.837778 5647 generic.go:356] "Generic (PLEG): container finished" podID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerID="f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826" exitCode=0 Nov 02 18:04:14.837920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.837825 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" Nov 02 18:04:14.837920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.837911 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" event={"ID":"78c37530-b9f9-432c-8115-1ab575ef0eef","Type":"ContainerDied","Data":"f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826"} Nov 02 18:04:14.838033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.837950 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l" event={"ID":"78c37530-b9f9-432c-8115-1ab575ef0eef","Type":"ContainerDied","Data":"709d908cd34bbc14b0b9bf65ca169a3444cad0b22c37f5514ecafe8a6b4a76df"} Nov 02 18:04:14.839589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.839563 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" event={"ID":"21237a61-6e71-48d7-b08e-dc6e4b3ebeab","Type":"ContainerStarted","Data":"604f953a06a9059e838323b0ad7008fac332b24e2516aa544e3f7f51bdedb24e"} Nov 02 18:04:14.849215 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.848435 5647 scope.go:122] "RemoveContainer" containerID="9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a" Nov 02 18:04:14.850343 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:14.850094 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a\": container with ID starting with 9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a not found: ID does not exist" containerID="9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a" Nov 02 18:04:14.850343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.850139 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a"} err="failed to get container status \"9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a\": rpc error: code = NotFound desc = could not find container \"9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a\": container with ID starting with 9882517158c99478091536995b9076aea79cd5719fef39a7987d4ba22d40de2a not found: ID does not exist" Nov 02 18:04:14.850343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.850166 5647 scope.go:122] "RemoveContainer" containerID="f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826" Nov 02 18:04:14.864234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.864208 5647 scope.go:122] "RemoveContainer" containerID="f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826" Nov 02 18:04:14.867187 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:14.867109 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826\": container with ID starting with f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826 not found: ID does not exist" containerID="f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826" Nov 02 18:04:14.867187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.867150 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826"} err="failed to get container status \"f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826\": rpc error: code = NotFound desc = could not find container \"f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826\": container with ID starting with f4ec1d06adbaddc096b99b5bded87358903f2468b8f4602ab3f93127416d6826 not found: ID does not exist" Nov 02 18:04:14.929511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.929482 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c37530-b9f9-432c-8115-1ab575ef0eef-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:14.929511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.929511 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcxkf\" (UniqueName: \"kubernetes.io/projected/78c37530-b9f9-432c-8115-1ab575ef0eef-kube-api-access-jcxkf\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:14.989589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:14.989547 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:15.031059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.030924 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca\") pod \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " Nov 02 18:04:15.031059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.031054 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca\") pod \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " Nov 02 18:04:15.031305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.031117 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5\" (UniqueName: \"kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5\") pod \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\" (UID: \"f6211150-1a7a-4f0e-a5bd-0172dea625b1\") " Nov 02 18:04:15.031450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.031301 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca" pod "f6211150-1a7a-4f0e-a5bd-0172dea625b1" (UID: "f6211150-1a7a-4f0e-a5bd-0172dea625b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:15.031761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.031608 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:15.033614 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.033586 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5" pod "f6211150-1a7a-4f0e-a5bd-0172dea625b1" (UID: "f6211150-1a7a-4f0e-a5bd-0172dea625b1"). InnerVolumeSpecName "kube-api-access-57rb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:15.033767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.033701 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca" pod "f6211150-1a7a-4f0e-a5bd-0172dea625b1" (UID: "f6211150-1a7a-4f0e-a5bd-0172dea625b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:15.082478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.082447 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9"] Nov 02 18:04:15.090188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.090154 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-943ca1-564f77749-fklf9"] Nov 02 18:04:15.132403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.132367 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6211150-1a7a-4f0e-a5bd-0172dea625b1-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:15.132403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.132403 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57rb5\" (UniqueName: \"kubernetes.io/projected/f6211150-1a7a-4f0e-a5bd-0172dea625b1-kube-api-access-57rb5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:15.202193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.202161 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l"] Nov 02 18:04:15.205829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.205800 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-943ca1-66dc8688d7-kmw9l"] Nov 02 18:04:15.848727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.848693 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" Nov 02 18:04:15.848727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.848713 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p" event={"ID":"f6211150-1a7a-4f0e-a5bd-0172dea625b1","Type":"ContainerDied","Data":"2c977b15f35ea15bb44f7a13c8a0c8fff1d5c8783ed124ac416f38703131c7d7"} Nov 02 18:04:15.849199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:15.848809 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c977b15f35ea15bb44f7a13c8a0c8fff1d5c8783ed124ac416f38703131c7d7" Nov 02 18:04:17.082670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:17.082639 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" path="/var/lib/kubelet/pods/66ba1257-b531-4d87-b7f8-e2982a7b24e6/volumes" Nov 02 18:04:17.083225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:17.083200 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" path="/var/lib/kubelet/pods/78c37530-b9f9-432c-8115-1ab575ef0eef/volumes" Nov 02 18:04:17.862304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:17.862263 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" event={"ID":"21237a61-6e71-48d7-b08e-dc6e4b3ebeab","Type":"ContainerStarted","Data":"a5359a68bf78250b46afbd79b6bde680e89adff4c1fe78511a0a5229e2a8ce1d"} Nov 02 18:04:17.862489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:17.862460 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:17.982836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:17.982782 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" podStartSLOduration=2.5544835729999997 podStartE2EDuration="4.982767293s" totalImagesPullingTime="2.42828372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:17.980694961 +0000 UTC m=+2445.054222165" watchObservedRunningTime="2026-11-02 18:04:17.982767293 +0000 UTC m=+2445.056294496" Nov 02 18:04:18.080521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:18.080490 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g"] Nov 02 18:04:18.080768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:18.080709 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" containerID="cri-o://11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca" gracePeriod=30 Nov 02 18:04:23.354320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.354256 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:04:23.431167 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431136 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431232 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431268 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431304 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431348 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431397 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431466 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431507 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.431641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431535 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket\") pod \"531cd99d-2559-464f-899d-88a9f483efe2\" (UID: \"531cd99d-2559-464f-899d-88a9f483efe2\") " Nov 02 18:04:23.432085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431715 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:23.432085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.431835 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:23.432277 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.432207 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:23.432449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.432416 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:23.432556 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.432447 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:23.432556 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.432526 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.432556 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.432547 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.432762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.432563 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.433711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.433686 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:04:23.433978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.433952 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "kube-api-access-rxz2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:23.434145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.434126 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:23.434245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.434142 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token" pod "531cd99d-2559-464f-899d-88a9f483efe2" (UID: "531cd99d-2559-464f-899d-88a9f483efe2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:23.533780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.533727 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/531cd99d-2559-464f-899d-88a9f483efe2-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.533780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.533779 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/531cd99d-2559-464f-899d-88a9f483efe2-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.534006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.533791 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.534006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.533803 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/531cd99d-2559-464f-899d-88a9f483efe2-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.534006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.533815 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxz2g\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-kube-api-access-rxz2g\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.534006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.533829 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/531cd99d-2559-464f-899d-88a9f483efe2-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:23.873790 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.873763 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:04:23.898617 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.898584 5647 generic.go:356] "Generic (PLEG): container finished" podID="531cd99d-2559-464f-899d-88a9f483efe2" containerID="11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca" exitCode=0 Nov 02 18:04:23.898861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.898632 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" Nov 02 18:04:23.898861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.898670 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" event={"ID":"531cd99d-2559-464f-899d-88a9f483efe2","Type":"ContainerDied","Data":"11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca"} Nov 02 18:04:23.898861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.898764 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g" event={"ID":"531cd99d-2559-464f-899d-88a9f483efe2","Type":"ContainerDied","Data":"af7766fbca067116953a177fc66f2e1a3c82819183c872cd873ef9930fb1c754"} Nov 02 18:04:23.898861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.898774 5647 scope.go:122] "RemoveContainer" containerID="11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca" Nov 02 18:04:23.909482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.909465 5647 scope.go:122] "RemoveContainer" containerID="ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c" Nov 02 18:04:23.919111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.919089 5647 scope.go:122] "RemoveContainer" containerID="11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca" Nov 02 18:04:23.919437 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:23.919417 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca\": container with ID starting with 11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca not found: ID does not exist" containerID="11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca" Nov 02 18:04:23.919497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.919449 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca"} err="failed to get container status \"11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca\": rpc error: code = NotFound desc = could not find container \"11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca\": container with ID starting with 11af1cb48b1cc9ccd06d8210035b97fa8cba06f196b53f0753a3dcbf446effca not found: ID does not exist" Nov 02 18:04:23.919497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.919467 5647 scope.go:122] "RemoveContainer" containerID="ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c" Nov 02 18:04:23.919763 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:23.919721 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c\": container with ID starting with ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c not found: ID does not exist" containerID="ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c" Nov 02 18:04:23.919876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:23.919769 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c"} err="failed to get container status \"ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c\": rpc error: code = NotFound desc = could not find container \"ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c\": container with ID starting with ce2300e58cc03af44635bc97d0bf1f22e2f9a22e85abe7fd29baba66fce64f3c not found: ID does not exist" Nov 02 18:04:24.114617 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:24.114579 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g"] Nov 02 18:04:24.120605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:24.120574 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-fd04fa-openshift-default-d4f876b9-tf66g"] Nov 02 18:04:25.081201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:25.081163 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="531cd99d-2559-464f-899d-88a9f483efe2" path="/var/lib/kubelet/pods/531cd99d-2559-464f-899d-88a9f483efe2/volumes" Nov 02 18:04:26.109048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:26.109011 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29"] Nov 02 18:04:28.320881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.320846 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l"] Nov 02 18:04:28.321291 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.321048 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" podUID="1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" containerName="maas-api" containerID="cri-o://2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66" gracePeriod=30 Nov 02 18:04:28.623927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.623891 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 18:04:28.796468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.796430 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8\" (UniqueName: \"kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8\") pod \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " Nov 02 18:04:28.796673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.796538 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\") pod \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " Nov 02 18:04:28.796673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.796603 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs\") pod \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " Nov 02 18:04:28.796673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.796654 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca\") pod \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\" (UID: \"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7\") " Nov 02 18:04:28.797120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.797094 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca" pod "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" (UID: "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:28.797349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.797284 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:28.798917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.798893 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls" pod "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" (UID: "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:28.799021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.798929 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8" pod "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" (UID: "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7"). InnerVolumeSpecName "kube-api-access-xmhz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:28.799021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.798966 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs" pod "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" (UID: "1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:28.898964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.898874 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:28.898964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.898904 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhz8\" (UniqueName: \"kubernetes.io/projected/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-kube-api-access-xmhz8\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:28.898964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.898915 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:28.933175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.933142 5647 generic.go:356] "Generic (PLEG): container finished" podID="1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" containerID="2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66" exitCode=0 Nov 02 18:04:28.933369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.933201 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" Nov 02 18:04:28.933369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.933238 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" event={"ID":"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7","Type":"ContainerDied","Data":"2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66"} Nov 02 18:04:28.933369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.933315 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l" event={"ID":"1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7","Type":"ContainerDied","Data":"b6cd3a9a08b2666d17b6ead34ff3801d3f201eda7b2f1671fe6284e3042e2e4c"} Nov 02 18:04:28.933508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.933362 5647 scope.go:122] "RemoveContainer" containerID="2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66" Nov 02 18:04:28.945357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.945335 5647 scope.go:122] "RemoveContainer" containerID="2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66" Nov 02 18:04:28.945711 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:28.945686 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66\": container with ID starting with 2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66 not found: ID does not exist" containerID="2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66" Nov 02 18:04:28.945827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:28.945724 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66"} err="failed to get container status \"2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66\": rpc error: code = NotFound desc = could not find container \"2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66\": container with ID starting with 2c454a11b81068158bac04130caf852a30e1bf1af5daa99cb7b5ecb806ff0d66 not found: ID does not exist" Nov 02 18:04:29.100033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:29.100003 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l"] Nov 02 18:04:29.104143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:29.104114 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-943ca1-b8f5fcb9b-d897l"] Nov 02 18:04:31.081960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.081923 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" path="/var/lib/kubelet/pods/1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7/volumes" Nov 02 18:04:31.110226 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.110191 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" podUID="45d1322d-d323-4c13-b50f-5cbbca395367" containerName="payload-pre-processing" containerID="cri-o://139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a" gracePeriod=25 Nov 02 18:04:31.380710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.380685 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:04:31.532276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.532237 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l\" (UniqueName: \"kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l\") pod \"45d1322d-d323-4c13-b50f-5cbbca395367\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " Nov 02 18:04:31.532485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.532458 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume\") pod \"45d1322d-d323-4c13-b50f-5cbbca395367\" (UID: \"45d1322d-d323-4c13-b50f-5cbbca395367\") " Nov 02 18:04:31.532951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.532923 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume" pod "45d1322d-d323-4c13-b50f-5cbbca395367" (UID: "45d1322d-d323-4c13-b50f-5cbbca395367"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:31.534675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.534646 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l" pod "45d1322d-d323-4c13-b50f-5cbbca395367" (UID: "45d1322d-d323-4c13-b50f-5cbbca395367"). InnerVolumeSpecName "kube-api-access-kl82l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:31.634479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.634390 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d1322d-d323-4c13-b50f-5cbbca395367-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:31.634479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.634419 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl82l\" (UniqueName: \"kubernetes.io/projected/45d1322d-d323-4c13-b50f-5cbbca395367-kube-api-access-kl82l\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:31.953829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.953714 5647 generic.go:356] "Generic (PLEG): container finished" podID="45d1322d-d323-4c13-b50f-5cbbca395367" containerID="139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a" exitCode=0 Nov 02 18:04:31.953829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.953788 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" Nov 02 18:04:31.953829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.953806 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" event={"ID":"45d1322d-d323-4c13-b50f-5cbbca395367","Type":"ContainerDied","Data":"139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a"} Nov 02 18:04:31.954127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.953873 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" event={"ID":"45d1322d-d323-4c13-b50f-5cbbca395367","Type":"ContainerDied","Data":"9f5d282b80fed5b8d654759f4d37faee8e5354ade8a0863aa34acbbce64bb115"} Nov 02 18:04:31.954127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.953904 5647 scope.go:122] "RemoveContainer" containerID="139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a" Nov 02 18:04:31.964007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.963985 5647 scope.go:122] "RemoveContainer" containerID="139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a" Nov 02 18:04:31.964312 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:31.964293 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a\": container with ID starting with 139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a not found: ID does not exist" containerID="139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a" Nov 02 18:04:31.964368 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:31.964320 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a"} err="failed to get container status \"139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a\": rpc error: code = NotFound desc = could not find container \"139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a\": container with ID starting with 139093fa7ba20727d95d55fec140f776919c14c925cfe4d90b0376f7dd5e1c2a not found: ID does not exist" Nov 02 18:04:32.122180 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:32.122143 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29"] Nov 02 18:04:32.171238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:32.171203 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29"] Nov 02 18:04:32.330092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:32.329986 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cf2b8a-5878cc4775-z8p29" podUID="45d1322d-d323-4c13-b50f-5cbbca395367" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.188:50052\" within 1s: context deadline exceeded" Nov 02 18:04:32.330092 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:04:32.330074 5647 logging.go:55] [core] [Channel #1721 SubChannel #1722]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.188:50052", ServerName: "10.134.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.188:50052: operation was canceled" Nov 02 18:04:33.081704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:33.081672 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d1322d-d323-4c13-b50f-5cbbca395367" path="/var/lib/kubelet/pods/45d1322d-d323-4c13-b50f-5cbbca395367/volumes" Nov 02 18:04:34.874882 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:34.874838 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:04:35.064401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:35.064367 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk"] Nov 02 18:04:35.064670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:35.064625 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" containerID="cri-o://2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2" gracePeriod=30 Nov 02 18:04:35.952378 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:35.952326 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.819414 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4"] Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820274 5647 cpu_manager.go:414] "containerMap: removing container" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820297 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820343 5647 cpu_manager.go:414] "containerMap: removing container" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerName="payload-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820352 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerName="payload-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820375 5647 cpu_manager.go:414] "containerMap: removing container" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerName="payload-pre-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820383 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerName="payload-pre-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820402 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f6211150-1a7a-4f0e-a5bd-0172dea625b1" containerName="revoke-keys" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820410 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6211150-1a7a-4f0e-a5bd-0172dea625b1" containerName="revoke-keys" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820439 5647 cpu_manager.go:414] "containerMap: removing container" podUID="45d1322d-d323-4c13-b50f-5cbbca395367" containerName="payload-pre-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820447 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d1322d-d323-4c13-b50f-5cbbca395367" containerName="payload-pre-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820472 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" containerName="maas-api" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820479 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" containerName="maas-api" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820626 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d1322d-d323-4c13-b50f-5cbbca395367" containerName="payload-pre-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820644 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820654 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee2ec0e-da4d-4b0d-ae19-812ddf147cc7" containerName="maas-api" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820667 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820680 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba1257-b531-4d87-b7f8-e2982a7b24e6" containerName="payload-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820692 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c37530-b9f9-432c-8115-1ab575ef0eef" containerName="payload-pre-processing" Nov 02 18:04:36.822771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.820704 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6211150-1a7a-4f0e-a5bd-0172dea625b1" containerName="revoke-keys" Nov 02 18:04:36.830829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.830797 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:36.876153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.876121 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4"] Nov 02 18:04:36.901787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.901717 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:36.901975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.901885 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t5gl\" (UniqueName: \"kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:36.902064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.902015 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:36.927535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.926996 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:36.927535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:36.927137 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:04:37.026760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.026402 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.026760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.026548 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.026760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.026616 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t5gl\" (UniqueName: \"kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.027690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.027654 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.029309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.029284 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.086671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.086640 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t5gl\" (UniqueName: \"kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl\") pod \"maas-api-revoke-sub-ef2e72bb-lx7s4\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.127216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.127163 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:04:37.127419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.127239 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:04:37.144873 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.144846 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:37.532371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.532115 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4"] Nov 02 18:04:37.535216 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:04:37.535188 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8529d7_ea1e_4f32_b3b2_49b2bea8a99e.slice/crio-6feb8da3591a41076020f6c2db77f23b18181bb544025864ce5786d69a8e1038 WatchSource:0}: Error finding container 6feb8da3591a41076020f6c2db77f23b18181bb544025864ce5786d69a8e1038: Status 404 returned error can't find the container with id 6feb8da3591a41076020f6c2db77f23b18181bb544025864ce5786d69a8e1038 Nov 02 18:04:37.996028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.995993 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" event={"ID":"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e","Type":"ContainerStarted","Data":"2a803b2ef96cc84e5db6e25f76589d49a0b8832a3750874de8fd03e7234d96f9"} Nov 02 18:04:37.996204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:37.996059 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" event={"ID":"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e","Type":"ContainerStarted","Data":"6feb8da3591a41076020f6c2db77f23b18181bb544025864ce5786d69a8e1038"} Nov 02 18:04:38.122513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:38.122446 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" podStartSLOduration=2.122426635 podStartE2EDuration="2.122426635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:38.11982968 +0000 UTC m=+2465.193356894" watchObservedRunningTime="2026-11-02 18:04:38.122426635 +0000 UTC m=+2465.195953841" Nov 02 18:04:39.025760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:39.024794 5647 generic.go:356] "Generic (PLEG): container finished" podID="cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" containerID="2a803b2ef96cc84e5db6e25f76589d49a0b8832a3750874de8fd03e7234d96f9" exitCode=0 Nov 02 18:04:39.025760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:39.024858 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" event={"ID":"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e","Type":"ContainerDied","Data":"2a803b2ef96cc84e5db6e25f76589d49a0b8832a3750874de8fd03e7234d96f9"} Nov 02 18:04:40.245286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.245258 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:40.364997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.364967 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca\") pod \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " Nov 02 18:04:40.365151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.365059 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl\" (UniqueName: \"kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl\") pod \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " Nov 02 18:04:40.365151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.365097 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca\") pod \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\" (UID: \"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e\") " Nov 02 18:04:40.365903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.365879 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca" pod "cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" (UID: "cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:40.368216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.368189 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca" pod "cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" (UID: "cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:40.368580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.368551 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl" pod "cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" (UID: "cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e"). InnerVolumeSpecName "kube-api-access-7t5gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:40.400436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.400406 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 18:04:40.465899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.465867 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.465922 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.465943 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466020 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466068 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466129 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466213 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466260 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466295 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:40.466346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466311 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo\") pod \"e06813a2-b541-443d-9415-ebcdc7a43438\" (UID: \"e06813a2-b541-443d-9415-ebcdc7a43438\") " Nov 02 18:04:40.466346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466319 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:40.466649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466425 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:40.466649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466539 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:40.466649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466557 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466774 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466790 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e06813a2-b541-443d-9415-ebcdc7a43438-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466806 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t5gl\" (UniqueName: \"kubernetes.io/projected/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-kube-api-access-7t5gl\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466817 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466829 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466841 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466849 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.466860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.466858 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.468571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.468543 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:40.468571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.468568 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "kube-api-access-hsst6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:40.468957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.468939 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:04:40.469025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.468951 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy" pod "e06813a2-b541-443d-9415-ebcdc7a43438" (UID: "e06813a2-b541-443d-9415-ebcdc7a43438"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:04:40.568714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.568680 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsst6\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-kube-api-access-hsst6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.568714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.568713 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e06813a2-b541-443d-9415-ebcdc7a43438-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.568967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.568725 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e06813a2-b541-443d-9415-ebcdc7a43438-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:40.568967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:40.568840 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e06813a2-b541-443d-9415-ebcdc7a43438-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:41.041839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.041784 5647 generic.go:356] "Generic (PLEG): container finished" podID="e06813a2-b541-443d-9415-ebcdc7a43438" containerID="2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2" exitCode=0 Nov 02 18:04:41.041839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.041851 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" Nov 02 18:04:41.042088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.041867 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" event={"ID":"e06813a2-b541-443d-9415-ebcdc7a43438","Type":"ContainerDied","Data":"2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2"} Nov 02 18:04:41.042088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.041930 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk" event={"ID":"e06813a2-b541-443d-9415-ebcdc7a43438","Type":"ContainerDied","Data":"45d1e5edd8458373907696d1b76218c430522ce0f6a0e181c4ffefceac962a9a"} Nov 02 18:04:41.042088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.041958 5647 scope.go:122] "RemoveContainer" containerID="2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2" Nov 02 18:04:41.043467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.043446 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" Nov 02 18:04:41.043559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.043472 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4" event={"ID":"cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e","Type":"ContainerDied","Data":"6feb8da3591a41076020f6c2db77f23b18181bb544025864ce5786d69a8e1038"} Nov 02 18:04:41.043559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.043523 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6feb8da3591a41076020f6c2db77f23b18181bb544025864ce5786d69a8e1038" Nov 02 18:04:41.052268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.052248 5647 scope.go:122] "RemoveContainer" containerID="f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746" Nov 02 18:04:41.061568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.061551 5647 scope.go:122] "RemoveContainer" containerID="2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2" Nov 02 18:04:41.061895 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:41.061870 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2\": container with ID starting with 2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2 not found: ID does not exist" containerID="2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2" Nov 02 18:04:41.061985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.061902 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2"} err="failed to get container status \"2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2\": rpc error: code = NotFound desc = could not find container \"2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2\": container with ID starting with 2d5fe02442e9d4e8b0ac1e42b83597838aa13136ce6801f0e2ec06ed00ba13d2 not found: ID does not exist" Nov 02 18:04:41.061985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.061920 5647 scope.go:122] "RemoveContainer" containerID="f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746" Nov 02 18:04:41.062207 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:41.062188 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746\": container with ID starting with f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746 not found: ID does not exist" containerID="f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746" Nov 02 18:04:41.062260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.062214 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746"} err="failed to get container status \"f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746\": rpc error: code = NotFound desc = could not find container \"f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746\": container with ID starting with f71c320d8a75031c092080f9388fd8ff26249809ca3bf3076902fa713372f746 not found: ID does not exist" Nov 02 18:04:41.269571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.269540 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk"] Nov 02 18:04:41.325845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:41.324590 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-943ca1-openshift-default-69bd9f8b4f-l6gqk"] Nov 02 18:04:42.590771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.590715 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5"] Nov 02 18:04:42.591377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591356 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" Nov 02 18:04:42.591377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591377 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" Nov 02 18:04:42.591514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591432 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" containerName="revoke-keys" Nov 02 18:04:42.591514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591438 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" containerName="revoke-keys" Nov 02 18:04:42.591514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591446 5647 cpu_manager.go:414] "containerMap: removing container" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" Nov 02 18:04:42.591514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591451 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="531cd99d-2559-464f-899d-88a9f483efe2" containerName="istio-proxy" Nov 02 18:04:42.591653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591538 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" Nov 02 18:04:42.591653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591549 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" containerName="revoke-keys" Nov 02 18:04:42.591653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.591557 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" Nov 02 18:04:42.596257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.596239 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.651643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.651600 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5"] Nov 02 18:04:42.699407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.699378 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:42.699581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.699507 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:04:42.703090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.703067 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.703172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.703123 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtln\" (UniqueName: \"kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.703294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.703277 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.804305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.804250 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtln\" (UniqueName: \"kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.804479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.804365 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.804525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.804485 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.805298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.805273 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.807011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.806983 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.863032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.862944 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtln\" (UniqueName: \"kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln\") pod \"maas-api-revoke-sub-768a2388-jw9j5\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:42.910715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:42.910674 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:43.083304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:43.083267 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" path="/var/lib/kubelet/pods/e06813a2-b541-443d-9415-ebcdc7a43438/volumes" Nov 02 18:04:43.290822 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:43.290787 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5"] Nov 02 18:04:43.292515 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:04:43.292481 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ad07d5_2919_497e_a811_5f3c9da8c84c.slice/crio-0a0bdb4b64a4aa05b131f7b18711ae666fc26f7948920306cd93f0c161a7524c WatchSource:0}: Error finding container 0a0bdb4b64a4aa05b131f7b18711ae666fc26f7948920306cd93f0c161a7524c: Status 404 returned error can't find the container with id 0a0bdb4b64a4aa05b131f7b18711ae666fc26f7948920306cd93f0c161a7524c Nov 02 18:04:44.066511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:44.066418 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" event={"ID":"62ad07d5-2919-497e-a811-5f3c9da8c84c","Type":"ContainerStarted","Data":"6f09f2155d0b1aa3d9dcd3a6b353d2d054ed9f158af3958221c776e5d2dadda7"} Nov 02 18:04:44.066511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:44.066492 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" event={"ID":"62ad07d5-2919-497e-a811-5f3c9da8c84c","Type":"ContainerStarted","Data":"0a0bdb4b64a4aa05b131f7b18711ae666fc26f7948920306cd93f0c161a7524c"} Nov 02 18:04:44.184501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:44.184451 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" podStartSLOduration=2.184431308 podStartE2EDuration="2.184431308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:44.183889158 +0000 UTC m=+2471.257416369" watchObservedRunningTime="2026-11-02 18:04:44.184431308 +0000 UTC m=+2471.257958512" Nov 02 18:04:45.073920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:45.073879 5647 generic.go:356] "Generic (PLEG): container finished" podID="62ad07d5-2919-497e-a811-5f3c9da8c84c" containerID="6f09f2155d0b1aa3d9dcd3a6b353d2d054ed9f158af3958221c776e5d2dadda7" exitCode=0 Nov 02 18:04:45.080423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:45.080391 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" event={"ID":"62ad07d5-2919-497e-a811-5f3c9da8c84c","Type":"ContainerDied","Data":"6f09f2155d0b1aa3d9dcd3a6b353d2d054ed9f158af3958221c776e5d2dadda7"} Nov 02 18:04:45.758486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:45.758452 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf"] Nov 02 18:04:45.889541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:45.888422 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r"] Nov 02 18:04:46.084366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.084269 5647 generic.go:356] "Generic (PLEG): container finished" podID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerID="979deb33e252e5b39dd1304aab8416b050249f73521bb8c9590af67686213b43" exitCode=0 Nov 02 18:04:46.084366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.084349 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" event={"ID":"080fb4bf-8385-4e94-a757-adf3b771f4fa","Type":"ContainerDied","Data":"979deb33e252e5b39dd1304aab8416b050249f73521bb8c9590af67686213b43"} Nov 02 18:04:46.085036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.084887 5647 scope.go:122] "RemoveContainer" containerID="979deb33e252e5b39dd1304aab8416b050249f73521bb8c9590af67686213b43" Nov 02 18:04:46.086458 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.086417 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:04:46.086552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.086490 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:04:46.086552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.086530 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:04:46.278458 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.276397 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:46.345463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.345423 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln\" (UniqueName: \"kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln\") pod \"62ad07d5-2919-497e-a811-5f3c9da8c84c\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " Nov 02 18:04:46.357676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.348311 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln" pod "62ad07d5-2919-497e-a811-5f3c9da8c84c" (UID: "62ad07d5-2919-497e-a811-5f3c9da8c84c"). InnerVolumeSpecName "kube-api-access-xrtln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:46.357676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.348552 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca\") pod \"62ad07d5-2919-497e-a811-5f3c9da8c84c\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " Nov 02 18:04:46.357676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.348673 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca\") pod \"62ad07d5-2919-497e-a811-5f3c9da8c84c\" (UID: \"62ad07d5-2919-497e-a811-5f3c9da8c84c\") " Nov 02 18:04:46.362072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.362029 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca" pod "62ad07d5-2919-497e-a811-5f3c9da8c84c" (UID: "62ad07d5-2919-497e-a811-5f3c9da8c84c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:46.370785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.362862 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrtln\" (UniqueName: \"kubernetes.io/projected/62ad07d5-2919-497e-a811-5f3c9da8c84c-kube-api-access-xrtln\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:46.370785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.362889 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:46.373196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.373162 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca" pod "62ad07d5-2919-497e-a811-5f3c9da8c84c" (UID: "62ad07d5-2919-497e-a811-5f3c9da8c84c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:04:46.464283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:46.464243 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ad07d5-2919-497e-a811-5f3c9da8c84c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:47.092186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.092159 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" Nov 02 18:04:47.092593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.092190 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5" event={"ID":"62ad07d5-2919-497e-a811-5f3c9da8c84c","Type":"ContainerDied","Data":"0a0bdb4b64a4aa05b131f7b18711ae666fc26f7948920306cd93f0c161a7524c"} Nov 02 18:04:47.092593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.092251 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a0bdb4b64a4aa05b131f7b18711ae666fc26f7948920306cd93f0c161a7524c" Nov 02 18:04:47.094492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.094465 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" event={"ID":"080fb4bf-8385-4e94-a757-adf3b771f4fa","Type":"ContainerStarted","Data":"7dba64d62b455cc06410a37f540fc4bfcf1050d56575a2a2cc2f489c289c81dc"} Nov 02 18:04:47.722759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.720127 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:04:47.732924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.732878 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:04:47.733111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:47.732948 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:04:48.381847 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:48.381807 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:04:48.705674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:48.705152 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:04:48.705674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:48.705227 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:04:49.726945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:49.726899 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:04:49.727393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:49.726984 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:04:50.716757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:50.716188 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:04:50.716757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:50.716267 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:04:50.758925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:50.758884 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" podUID="da114dd0-7138-4f17-b68c-e8c95e4a336e" containerName="payload-processing" containerID="cri-o://06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356" gracePeriod=25 Nov 02 18:04:50.888984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:50.888943 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" podUID="9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" containerName="payload-pre-processing" containerID="cri-o://b318e2591c979d72476f7c82e05004769c25625a2d3831e4499298c4b6bf5fa4" gracePeriod=25 Nov 02 18:04:51.092173 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.092146 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 18:04:51.142561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.141823 5647 generic.go:356] "Generic (PLEG): container finished" podID="9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" containerID="b318e2591c979d72476f7c82e05004769c25625a2d3831e4499298c4b6bf5fa4" exitCode=0 Nov 02 18:04:51.142561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.141896 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" event={"ID":"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40","Type":"ContainerDied","Data":"b318e2591c979d72476f7c82e05004769c25625a2d3831e4499298c4b6bf5fa4"} Nov 02 18:04:51.143309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.143289 5647 generic.go:356] "Generic (PLEG): container finished" podID="da114dd0-7138-4f17-b68c-e8c95e4a336e" containerID="06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356" exitCode=0 Nov 02 18:04:51.143439 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.143424 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" event={"ID":"da114dd0-7138-4f17-b68c-e8c95e4a336e","Type":"ContainerDied","Data":"06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356"} Nov 02 18:04:51.143484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.143458 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" event={"ID":"da114dd0-7138-4f17-b68c-e8c95e4a336e","Type":"ContainerDied","Data":"fed4eeda8b08fd6e08355f2e2efadd762406a0bb4c7d89faa9da18c837504e99"} Nov 02 18:04:51.143484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.143479 5647 scope.go:122] "RemoveContainer" containerID="06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356" Nov 02 18:04:51.143601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.143591 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf" Nov 02 18:04:51.159640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.159614 5647 scope.go:122] "RemoveContainer" containerID="06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356" Nov 02 18:04:51.160330 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:04:51.160269 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356\": container with ID starting with 06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356 not found: ID does not exist" containerID="06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356" Nov 02 18:04:51.160500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.160307 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356"} err="failed to get container status \"06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356\": rpc error: code = NotFound desc = could not find container \"06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356\": container with ID starting with 06617c57923d9f96c503f37915bce0f5fec9f80d429ae297f94ad958bdd5e356 not found: ID does not exist" Nov 02 18:04:51.267071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.267043 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 18:04:51.267448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.267431 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz\" (UniqueName: \"kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz\") pod \"da114dd0-7138-4f17-b68c-e8c95e4a336e\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " Nov 02 18:04:51.267515 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.267492 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume\") pod \"da114dd0-7138-4f17-b68c-e8c95e4a336e\" (UID: \"da114dd0-7138-4f17-b68c-e8c95e4a336e\") " Nov 02 18:04:51.267941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.267900 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume" pod "da114dd0-7138-4f17-b68c-e8c95e4a336e" (UID: "da114dd0-7138-4f17-b68c-e8c95e4a336e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:51.269750 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.269717 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz" pod "da114dd0-7138-4f17-b68c-e8c95e4a336e" (UID: "da114dd0-7138-4f17-b68c-e8c95e4a336e"). InnerVolumeSpecName "kube-api-access-vbgnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.327828 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86"] Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328348 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" containerName="payload-pre-processing" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328361 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" containerName="payload-pre-processing" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328388 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328393 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06813a2-b541-443d-9415-ebcdc7a43438" containerName="istio-proxy" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328401 5647 cpu_manager.go:414] "containerMap: removing container" podUID="62ad07d5-2919-497e-a811-5f3c9da8c84c" containerName="revoke-keys" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328406 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ad07d5-2919-497e-a811-5f3c9da8c84c" containerName="revoke-keys" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328432 5647 cpu_manager.go:414] "containerMap: removing container" podUID="da114dd0-7138-4f17-b68c-e8c95e4a336e" containerName="payload-processing" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328438 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da114dd0-7138-4f17-b68c-e8c95e4a336e" containerName="payload-processing" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328529 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" containerName="payload-pre-processing" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328542 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ad07d5-2919-497e-a811-5f3c9da8c84c" containerName="revoke-keys" Nov 02 18:04:51.328856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.328552 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="da114dd0-7138-4f17-b68c-e8c95e4a336e" containerName="payload-processing" Nov 02 18:04:51.331817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.331798 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.369078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.369046 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume\") pod \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " Nov 02 18:04:51.369273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.369252 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh\" (UniqueName: \"kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh\") pod \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\" (UID: \"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40\") " Nov 02 18:04:51.369512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.369485 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume" pod "9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" (UID: "9fa1b65b-9b54-41cc-b17f-0d0b9e823f40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:04:51.369633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.369616 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da114dd0-7138-4f17-b68c-e8c95e4a336e-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:51.369690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.369640 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:51.369690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.369655 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbgnz\" (UniqueName: \"kubernetes.io/projected/da114dd0-7138-4f17-b68c-e8c95e4a336e-kube-api-access-vbgnz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:51.371950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.371909 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh" pod "9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" (UID: "9fa1b65b-9b54-41cc-b17f-0d0b9e823f40"). InnerVolumeSpecName "kube-api-access-ntxjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:04:51.457779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.448994 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86"] Nov 02 18:04:51.470627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.470582 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.470819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.470711 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.470819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.470804 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.470964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.470907 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwzgq\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.470964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.470954 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.471086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.471030 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.471086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.471062 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.471187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.471118 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.471351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.471239 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.471351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.471329 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntxjh\" (UniqueName: \"kubernetes.io/projected/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40-kube-api-access-ntxjh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:04:51.572861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.572827 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.572877 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.572924 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.572940 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.572986 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwzgq\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573014 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573040 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573059 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573090 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573292 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.573424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573289 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.574011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573774 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.574011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573810 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.574011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.573975 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.575891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.575862 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.576249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.576227 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.639683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.635296 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.639683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.636235 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwzgq\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.644672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.644646 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:51.647015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.646990 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf"] Nov 02 18:04:51.652953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.652903 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2289f-65dff5d58c-j27xf"] Nov 02 18:04:51.708764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.706189 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:04:51.708764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:51.706249 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:04:52.118290 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.118252 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86"] Nov 02 18:04:52.163260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.163231 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" event={"ID":"9fa1b65b-9b54-41cc-b17f-0d0b9e823f40","Type":"ContainerDied","Data":"78fb5565181b2f951b9549431e42ce7c8860bd6d731455531df828a37c73652c"} Nov 02 18:04:52.163577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.163552 5647 scope.go:122] "RemoveContainer" containerID="b318e2591c979d72476f7c82e05004769c25625a2d3831e4499298c4b6bf5fa4" Nov 02 18:04:52.165174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.163484 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r" Nov 02 18:04:52.170131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.169055 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:04:52.170131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.169145 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:04:52.170131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.169190 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:04:52.362759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.360169 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r"] Nov 02 18:04:52.365295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.364947 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2289f-58c57cc78c-fjm2r"] Nov 02 18:04:52.727760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.721627 5647 patch_prober.go:29] interesting pod/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Nov 02 18:04:52.727760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:52.721696 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Nov 02 18:04:53.086946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.085847 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa1b65b-9b54-41cc-b17f-0d0b9e823f40" path="/var/lib/kubelet/pods/9fa1b65b-9b54-41cc-b17f-0d0b9e823f40/volumes" Nov 02 18:04:53.086946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.086425 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da114dd0-7138-4f17-b68c-e8c95e4a336e" path="/var/lib/kubelet/pods/da114dd0-7138-4f17-b68c-e8c95e4a336e/volumes" Nov 02 18:04:53.176231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.176177 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" event={"ID":"c6928b40-6f6a-4f4f-890b-4acccb138cc6","Type":"ContainerStarted","Data":"ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0"} Nov 02 18:04:53.176789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.176256 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" event={"ID":"c6928b40-6f6a-4f4f-890b-4acccb138cc6","Type":"ContainerStarted","Data":"0fd5ded19a1216644b830952fd966a53a1ba7e13246754f50090d639e04f1b78"} Nov 02 18:04:53.300629 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.300578 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podStartSLOduration=2.300565276 podStartE2EDuration="2.300565276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:04:53.29978818 +0000 UTC m=+2480.373315383" watchObservedRunningTime="2026-11-02 18:04:53.300565276 +0000 UTC m=+2480.374092546" Nov 02 18:04:53.645918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.645882 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:53.647562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.647529 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 02 18:04:53.647727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.647579 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 02 18:04:53.727218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.727183 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:04:53.729292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:53.729253 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:04:54.645274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:54.645234 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 02 18:04:54.645694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:54.645295 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 02 18:04:55.645538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:55.645493 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 02 18:04:55.646007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:55.645560 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 02 18:04:56.645581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:56.645546 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 02 18:04:56.646009 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:56.645599 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 02 18:04:57.646019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:57.645977 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 02 18:04:57.646446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:57.646038 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 02 18:04:58.359744 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.359693 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m"] Nov 02 18:04:58.363593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.363574 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.380940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.380912 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.381100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.380953 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfb6r\" (UniqueName: \"kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.381155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.381114 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.428788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.418598 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:04:58.468774 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.468713 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:04:58.471199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.471173 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m"] Nov 02 18:04:58.481868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.481842 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.482019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.481927 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.482019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.481948 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfb6r\" (UniqueName: \"kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.482634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.482606 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.484401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.484382 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.540627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.540586 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfb6r\" (UniqueName: \"kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r\") pod \"maas-api-revoke-sub-52e38ad7-rjq6m\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:58.645860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.645767 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 02 18:04:58.645860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.645832 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 02 18:04:58.676189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:58.676152 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:04:59.054673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:59.054647 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m"] Nov 02 18:04:59.056386 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:04:59.056355 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc973c05_8111_4239_a679_8e2ed571d3e9.slice/crio-1936055b4228eb011e9e4b12fbdd888a65ec4841c7ce9740158da99c1acbf43b WatchSource:0}: Error finding container 1936055b4228eb011e9e4b12fbdd888a65ec4841c7ce9740158da99c1acbf43b: Status 404 returned error can't find the container with id 1936055b4228eb011e9e4b12fbdd888a65ec4841c7ce9740158da99c1acbf43b Nov 02 18:04:59.218456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:59.218418 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" event={"ID":"cc973c05-8111-4239-a679-8e2ed571d3e9","Type":"ContainerStarted","Data":"1936055b4228eb011e9e4b12fbdd888a65ec4841c7ce9740158da99c1acbf43b"} Nov 02 18:04:59.872834 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:59.872781 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:59.873319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:59.873293 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:04:59.874194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:04:59.874176 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:05:00.226346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:00.226226 5647 generic.go:356] "Generic (PLEG): container finished" podID="cc973c05-8111-4239-a679-8e2ed571d3e9" containerID="fc55e047d0a68682d686902c49224e4e1aaac3052105133960e56f406609dc99" exitCode=0 Nov 02 18:05:00.226346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:00.226306 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" event={"ID":"cc973c05-8111-4239-a679-8e2ed571d3e9","Type":"ContainerDied","Data":"fc55e047d0a68682d686902c49224e4e1aaac3052105133960e56f406609dc99"} Nov 02 18:05:01.378331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.378308 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:05:01.413716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.413682 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca\") pod \"cc973c05-8111-4239-a679-8e2ed571d3e9\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " Nov 02 18:05:01.413904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.413792 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca\") pod \"cc973c05-8111-4239-a679-8e2ed571d3e9\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " Nov 02 18:05:01.413904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.413827 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r\" (UniqueName: \"kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r\") pod \"cc973c05-8111-4239-a679-8e2ed571d3e9\" (UID: \"cc973c05-8111-4239-a679-8e2ed571d3e9\") " Nov 02 18:05:01.414182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.414152 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca" pod "cc973c05-8111-4239-a679-8e2ed571d3e9" (UID: "cc973c05-8111-4239-a679-8e2ed571d3e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:01.415939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.415902 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca" pod "cc973c05-8111-4239-a679-8e2ed571d3e9" (UID: "cc973c05-8111-4239-a679-8e2ed571d3e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:01.416060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.416028 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r" pod "cc973c05-8111-4239-a679-8e2ed571d3e9" (UID: "cc973c05-8111-4239-a679-8e2ed571d3e9"). InnerVolumeSpecName "kube-api-access-zfb6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:01.515137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.515062 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:01.515137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.515090 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc973c05-8111-4239-a679-8e2ed571d3e9-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:01.515137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:01.515101 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfb6r\" (UniqueName: \"kubernetes.io/projected/cc973c05-8111-4239-a679-8e2ed571d3e9-kube-api-access-zfb6r\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:02.238830 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:02.238803 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" Nov 02 18:05:02.239018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:02.238824 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m" event={"ID":"cc973c05-8111-4239-a679-8e2ed571d3e9","Type":"ContainerDied","Data":"1936055b4228eb011e9e4b12fbdd888a65ec4841c7ce9740158da99c1acbf43b"} Nov 02 18:05:02.239018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:02.238878 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1936055b4228eb011e9e4b12fbdd888a65ec4841c7ce9740158da99c1acbf43b" Nov 02 18:05:05.947037 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:05:05.946994 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:05:16.897694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:16.897646 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk"] Nov 02 18:05:17.007276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:17.007232 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs"] Nov 02 18:05:17.628244 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:17.628204 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:05:20.983206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:20.983168 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59"] Nov 02 18:05:20.983799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:20.983728 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cc973c05-8111-4239-a679-8e2ed571d3e9" containerName="revoke-keys" Nov 02 18:05:20.983799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:20.983777 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc973c05-8111-4239-a679-8e2ed571d3e9" containerName="revoke-keys" Nov 02 18:05:20.983936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:20.983900 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc973c05-8111-4239-a679-8e2ed571d3e9" containerName="revoke-keys" Nov 02 18:05:20.988239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:20.988224 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.058409 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.058373 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd7ml\" (UniqueName: \"kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.058613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.058488 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.058613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.058559 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.093836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.093805 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:05:21.093836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.093805 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:05:21.096011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.095991 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59"] Nov 02 18:05:21.160064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.160026 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd7ml\" (UniqueName: \"kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.160229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.160128 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.160229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.160216 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.160908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.160890 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.162649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.162631 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.220059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.220023 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd7ml\" (UniqueName: \"kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml\") pod \"maas-api-revoke-sub-d53c5e4c-2cj59\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.300989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.300913 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:21.683818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.683791 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59"] Nov 02 18:05:21.685842 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:21.685801 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00119482_26d5_4ebb_a3b7_6942e73fa5cb.slice/crio-7a3a638d7a91c81f2ba7badcd72abfbe22ab41be8ed59560ac4092b0c991f4e1 WatchSource:0}: Error finding container 7a3a638d7a91c81f2ba7badcd72abfbe22ab41be8ed59560ac4092b0c991f4e1: Status 404 returned error can't find the container with id 7a3a638d7a91c81f2ba7badcd72abfbe22ab41be8ed59560ac4092b0c991f4e1 Nov 02 18:05:21.898719 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:21.898675 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" podUID="887fa935-2841-49f7-ad11-c08efd7283a8" containerName="payload-processing" containerID="cri-o://250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429" gracePeriod=25 Nov 02 18:05:22.007978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.007928 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" podUID="15e7997f-5dc9-42f3-bedc-5cb477431614" containerName="payload-pre-processing" containerID="cri-o://bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561" gracePeriod=25 Nov 02 18:05:22.158894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.158867 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:05:22.274469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.274379 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume\") pod \"887fa935-2841-49f7-ad11-c08efd7283a8\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " Nov 02 18:05:22.274469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.274441 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc\" (UniqueName: \"kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc\") pod \"887fa935-2841-49f7-ad11-c08efd7283a8\" (UID: \"887fa935-2841-49f7-ad11-c08efd7283a8\") " Nov 02 18:05:22.274773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.274713 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume" pod "887fa935-2841-49f7-ad11-c08efd7283a8" (UID: "887fa935-2841-49f7-ad11-c08efd7283a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:22.280928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.280890 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc" pod "887fa935-2841-49f7-ad11-c08efd7283a8" (UID: "887fa935-2841-49f7-ad11-c08efd7283a8"). InnerVolumeSpecName "kube-api-access-8chrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:22.327477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.327453 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:05:22.376374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.376340 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/887fa935-2841-49f7-ad11-c08efd7283a8-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:22.376374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.376369 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chrc\" (UniqueName: \"kubernetes.io/projected/887fa935-2841-49f7-ad11-c08efd7283a8-kube-api-access-8chrc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:22.385080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.385039 5647 generic.go:356] "Generic (PLEG): container finished" podID="15e7997f-5dc9-42f3-bedc-5cb477431614" containerID="bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561" exitCode=0 Nov 02 18:05:22.385246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.385091 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" Nov 02 18:05:22.385246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.385121 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" event={"ID":"15e7997f-5dc9-42f3-bedc-5cb477431614","Type":"ContainerDied","Data":"bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561"} Nov 02 18:05:22.385246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.385192 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs" event={"ID":"15e7997f-5dc9-42f3-bedc-5cb477431614","Type":"ContainerDied","Data":"a076f3af6443f73eda6311a80c64e43b81298b259f14735af2474a225104cf49"} Nov 02 18:05:22.385246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.385230 5647 scope.go:122] "RemoveContainer" containerID="bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561" Nov 02 18:05:22.387067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.387045 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" event={"ID":"00119482-26d5-4ebb-a3b7-6942e73fa5cb","Type":"ContainerStarted","Data":"5e8aba4e9dcbae9398a52efe519f31ae2d74317c29dd25d55a0d038068e214a9"} Nov 02 18:05:22.387199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.387092 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" event={"ID":"00119482-26d5-4ebb-a3b7-6942e73fa5cb","Type":"ContainerStarted","Data":"7a3a638d7a91c81f2ba7badcd72abfbe22ab41be8ed59560ac4092b0c991f4e1"} Nov 02 18:05:22.388490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.388471 5647 generic.go:356] "Generic (PLEG): container finished" podID="887fa935-2841-49f7-ad11-c08efd7283a8" containerID="250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429" exitCode=0 Nov 02 18:05:22.388566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.388521 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" Nov 02 18:05:22.388625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.388522 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" event={"ID":"887fa935-2841-49f7-ad11-c08efd7283a8","Type":"ContainerDied","Data":"250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429"} Nov 02 18:05:22.388673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.388643 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk" event={"ID":"887fa935-2841-49f7-ad11-c08efd7283a8","Type":"ContainerDied","Data":"fc3c7d3298299715670d0458fec349251ad66009ee237f8df90e0034912a19eb"} Nov 02 18:05:22.395194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.395175 5647 scope.go:122] "RemoveContainer" containerID="bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561" Nov 02 18:05:22.395505 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:05:22.395482 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561\": container with ID starting with bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561 not found: ID does not exist" containerID="bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561" Nov 02 18:05:22.395604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.395513 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561"} err="failed to get container status \"bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561\": rpc error: code = NotFound desc = could not find container \"bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561\": container with ID starting with bb14212e3f1c1be7b4d858eaf26ac7bf27e219ab289b5c33c292d863af111561 not found: ID does not exist" Nov 02 18:05:22.395604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.395532 5647 scope.go:122] "RemoveContainer" containerID="250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429" Nov 02 18:05:22.404494 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.404466 5647 scope.go:122] "RemoveContainer" containerID="250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429" Nov 02 18:05:22.404836 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:05:22.404812 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429\": container with ID starting with 250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429 not found: ID does not exist" containerID="250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429" Nov 02 18:05:22.404905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.404845 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429"} err="failed to get container status \"250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429\": rpc error: code = NotFound desc = could not find container \"250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429\": container with ID starting with 250550b99e9de2e6271b689858ac69b362299702538e4a761ddee705bedd2429 not found: ID does not exist" Nov 02 18:05:22.477039 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.477004 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg\" (UniqueName: \"kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg\") pod \"15e7997f-5dc9-42f3-bedc-5cb477431614\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " Nov 02 18:05:22.477231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.477116 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume\") pod \"15e7997f-5dc9-42f3-bedc-5cb477431614\" (UID: \"15e7997f-5dc9-42f3-bedc-5cb477431614\") " Nov 02 18:05:22.477728 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.477547 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume" pod "15e7997f-5dc9-42f3-bedc-5cb477431614" (UID: "15e7997f-5dc9-42f3-bedc-5cb477431614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:22.479393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.479370 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg" pod "15e7997f-5dc9-42f3-bedc-5cb477431614" (UID: "15e7997f-5dc9-42f3-bedc-5cb477431614"). InnerVolumeSpecName "kube-api-access-v7wrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:22.562160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.562058 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" podStartSLOduration=2.562043891 podStartE2EDuration="2.562043891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:22.561443025 +0000 UTC m=+2509.634970228" watchObservedRunningTime="2026-11-02 18:05:22.562043891 +0000 UTC m=+2509.635571094" Nov 02 18:05:22.578500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.578474 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7wrg\" (UniqueName: \"kubernetes.io/projected/15e7997f-5dc9-42f3-bedc-5cb477431614-kube-api-access-v7wrg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:22.578500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.578499 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e7997f-5dc9-42f3-bedc-5cb477431614-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:22.730127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.730092 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk"] Nov 02 18:05:22.737903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.737872 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-948451-545ff744df-2qbpk"] Nov 02 18:05:22.850284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.850234 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs"] Nov 02 18:05:22.855069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:22.855039 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-948451-7bb78f655-7bkqs"] Nov 02 18:05:23.081930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:23.081899 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e7997f-5dc9-42f3-bedc-5cb477431614" path="/var/lib/kubelet/pods/15e7997f-5dc9-42f3-bedc-5cb477431614/volumes" Nov 02 18:05:23.082355 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:23.082341 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887fa935-2841-49f7-ad11-c08efd7283a8" path="/var/lib/kubelet/pods/887fa935-2841-49f7-ad11-c08efd7283a8/volumes" Nov 02 18:05:23.397646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:23.397614 5647 generic.go:356] "Generic (PLEG): container finished" podID="00119482-26d5-4ebb-a3b7-6942e73fa5cb" containerID="5e8aba4e9dcbae9398a52efe519f31ae2d74317c29dd25d55a0d038068e214a9" exitCode=0 Nov 02 18:05:23.397898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:23.397704 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" event={"ID":"00119482-26d5-4ebb-a3b7-6942e73fa5cb","Type":"ContainerDied","Data":"5e8aba4e9dcbae9398a52efe519f31ae2d74317c29dd25d55a0d038068e214a9"} Nov 02 18:05:24.557778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.557748 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:24.710758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.710636 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca\") pod \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " Nov 02 18:05:24.711030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.710871 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca\") pod \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " Nov 02 18:05:24.711030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.710918 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml\" (UniqueName: \"kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml\") pod \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\" (UID: \"00119482-26d5-4ebb-a3b7-6942e73fa5cb\") " Nov 02 18:05:24.711263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.711233 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca" pod "00119482-26d5-4ebb-a3b7-6942e73fa5cb" (UID: "00119482-26d5-4ebb-a3b7-6942e73fa5cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:24.713019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.712991 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca" pod "00119482-26d5-4ebb-a3b7-6942e73fa5cb" (UID: "00119482-26d5-4ebb-a3b7-6942e73fa5cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:24.713203 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.713185 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml" pod "00119482-26d5-4ebb-a3b7-6942e73fa5cb" (UID: "00119482-26d5-4ebb-a3b7-6942e73fa5cb"). InnerVolumeSpecName "kube-api-access-zd7ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:24.812141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.812106 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:24.812141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.812136 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd7ml\" (UniqueName: \"kubernetes.io/projected/00119482-26d5-4ebb-a3b7-6942e73fa5cb-kube-api-access-zd7ml\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:24.812141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:24.812148 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00119482-26d5-4ebb-a3b7-6942e73fa5cb-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:25.411389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:25.411297 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" event={"ID":"00119482-26d5-4ebb-a3b7-6942e73fa5cb","Type":"ContainerDied","Data":"7a3a638d7a91c81f2ba7badcd72abfbe22ab41be8ed59560ac4092b0c991f4e1"} Nov 02 18:05:25.411389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:25.411351 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a3a638d7a91c81f2ba7badcd72abfbe22ab41be8ed59560ac4092b0c991f4e1" Nov 02 18:05:25.411389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:25.411313 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59" Nov 02 18:05:28.386200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.386164 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7"] Nov 02 18:05:28.386640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.386420 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" podUID="03d5b318-a3d4-456d-8eda-3c7951b1b426" containerName="maas-api" containerID="cri-o://a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4" gracePeriod=30 Nov 02 18:05:28.676619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.676592 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 18:05:28.864971 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.864938 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr\" (UniqueName: \"kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr\") pod \"03d5b318-a3d4-456d-8eda-3c7951b1b426\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " Nov 02 18:05:28.865193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.865095 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs\") pod \"03d5b318-a3d4-456d-8eda-3c7951b1b426\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " Nov 02 18:05:28.865193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.865149 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\") pod \"03d5b318-a3d4-456d-8eda-3c7951b1b426\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " Nov 02 18:05:28.865322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.865225 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca\") pod \"03d5b318-a3d4-456d-8eda-3c7951b1b426\" (UID: \"03d5b318-a3d4-456d-8eda-3c7951b1b426\") " Nov 02 18:05:28.865672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.865638 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca" pod "03d5b318-a3d4-456d-8eda-3c7951b1b426" (UID: "03d5b318-a3d4-456d-8eda-3c7951b1b426"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:28.867166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.867139 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr" pod "03d5b318-a3d4-456d-8eda-3c7951b1b426" (UID: "03d5b318-a3d4-456d-8eda-3c7951b1b426"). InnerVolumeSpecName "kube-api-access-cnbgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:28.867268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.867245 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs" pod "03d5b318-a3d4-456d-8eda-3c7951b1b426" (UID: "03d5b318-a3d4-456d-8eda-3c7951b1b426"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:28.867364 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.867349 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls" pod "03d5b318-a3d4-456d-8eda-3c7951b1b426" (UID: "03d5b318-a3d4-456d-8eda-3c7951b1b426"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:28.966572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.966482 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.966572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.966513 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03d5b318-a3d4-456d-8eda-3c7951b1b426-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.966572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.966524 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03d5b318-a3d4-456d-8eda-3c7951b1b426-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:28.966572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:28.966535 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnbgr\" (UniqueName: \"kubernetes.io/projected/03d5b318-a3d4-456d-8eda-3c7951b1b426-kube-api-access-cnbgr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:29.437068 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.437029 5647 generic.go:356] "Generic (PLEG): container finished" podID="03d5b318-a3d4-456d-8eda-3c7951b1b426" containerID="a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4" exitCode=0 Nov 02 18:05:29.437537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.437085 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" Nov 02 18:05:29.437537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.437113 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" event={"ID":"03d5b318-a3d4-456d-8eda-3c7951b1b426","Type":"ContainerDied","Data":"a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4"} Nov 02 18:05:29.437537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.437170 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7" event={"ID":"03d5b318-a3d4-456d-8eda-3c7951b1b426","Type":"ContainerDied","Data":"fed5971bc7abfac8ed0c0e4b09970a7403018777cbc8510085612e5429eb6493"} Nov 02 18:05:29.437537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.437191 5647 scope.go:122] "RemoveContainer" containerID="a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4" Nov 02 18:05:29.449083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.449063 5647 scope.go:122] "RemoveContainer" containerID="a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4" Nov 02 18:05:29.449399 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:05:29.449370 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4\": container with ID starting with a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4 not found: ID does not exist" containerID="a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4" Nov 02 18:05:29.449451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.449410 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4"} err="failed to get container status \"a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4\": rpc error: code = NotFound desc = could not find container \"a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4\": container with ID starting with a80e61bf96d957af19fd905080018e2e11aa04ed7c4698a8f2aac56dc43c16d4 not found: ID does not exist" Nov 02 18:05:29.606386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.606352 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7"] Nov 02 18:05:29.609538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:29.609508 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f2289f-784dfb5695-jftt7"] Nov 02 18:05:31.080808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:31.080774 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d5b318-a3d4-456d-8eda-3c7951b1b426" path="/var/lib/kubelet/pods/03d5b318-a3d4-456d-8eda-3c7951b1b426/volumes" Nov 02 18:05:34.928377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928331 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2"] Nov 02 18:05:34.928928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928907 5647 cpu_manager.go:414] "containerMap: removing container" podUID="887fa935-2841-49f7-ad11-c08efd7283a8" containerName="payload-processing" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928932 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887fa935-2841-49f7-ad11-c08efd7283a8" containerName="payload-processing" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928966 5647 cpu_manager.go:414] "containerMap: removing container" podUID="15e7997f-5dc9-42f3-bedc-5cb477431614" containerName="payload-pre-processing" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928974 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e7997f-5dc9-42f3-bedc-5cb477431614" containerName="payload-pre-processing" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928985 5647 cpu_manager.go:414] "containerMap: removing container" podUID="00119482-26d5-4ebb-a3b7-6942e73fa5cb" containerName="revoke-keys" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928990 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00119482-26d5-4ebb-a3b7-6942e73fa5cb" containerName="revoke-keys" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.928997 5647 cpu_manager.go:414] "containerMap: removing container" podUID="03d5b318-a3d4-456d-8eda-3c7951b1b426" containerName="maas-api" Nov 02 18:05:34.929003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.929001 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d5b318-a3d4-456d-8eda-3c7951b1b426" containerName="maas-api" Nov 02 18:05:34.929253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.929101 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="887fa935-2841-49f7-ad11-c08efd7283a8" containerName="payload-processing" Nov 02 18:05:34.929253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.929111 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e7997f-5dc9-42f3-bedc-5cb477431614" containerName="payload-pre-processing" Nov 02 18:05:34.929253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.929116 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="00119482-26d5-4ebb-a3b7-6942e73fa5cb" containerName="revoke-keys" Nov 02 18:05:34.929253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.929122 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d5b318-a3d4-456d-8eda-3c7951b1b426" containerName="maas-api" Nov 02 18:05:34.933863 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.933842 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:34.990239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:34.990197 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2"] Nov 02 18:05:35.034856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.034819 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pk8w\" (UniqueName: \"kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.034856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.034870 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.035092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.034988 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.038205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.038186 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:05:35.038267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.038257 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:05:35.136094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.136054 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pk8w\" (UniqueName: \"kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.136283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.136108 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.136283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.136171 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.136981 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.136954 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.138624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.138597 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.195164 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.195096 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pk8w\" (UniqueName: \"kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w\") pod \"maas-api-revoke-sub-27e10ec1-lzbv2\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.247329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.247290 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:35.388956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.388922 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg"] Nov 02 18:05:35.392973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.392957 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.440109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.440073 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td584\" (UniqueName: \"kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.440109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.440112 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.440347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.440206 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.456822 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.456727 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg"] Nov 02 18:05:35.541139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.541106 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td584\" (UniqueName: \"kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.541139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.541142 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.541401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.541174 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.541821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.541801 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.543569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.543548 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.600573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.600542 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td584\" (UniqueName: \"kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584\") pod \"maas-api-revoke-sub-1bb342c7-ntrmg\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.628074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.628048 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2"] Nov 02 18:05:35.629997 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:35.629960 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793def51_6fec_4b5a_98d4_a4b0c33067a8.slice/crio-c6d7a9c4bf64a88799c1a209c80a890e47e85985734239fc8b50f1fce2cc5ea3 WatchSource:0}: Error finding container c6d7a9c4bf64a88799c1a209c80a890e47e85985734239fc8b50f1fce2cc5ea3: Status 404 returned error can't find the container with id c6d7a9c4bf64a88799c1a209c80a890e47e85985734239fc8b50f1fce2cc5ea3 Nov 02 18:05:35.706603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:35.706570 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:35.946639 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:05:35.946607 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:05:36.086274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.086245 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg"] Nov 02 18:05:36.087549 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:36.087522 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0bdbc45_6cde_4466_b68a_4dcb6721c4d6.slice/crio-8cf896445229935607fff166132d6dc0d26e02f6f7138d776b224a123b2193ff WatchSource:0}: Error finding container 8cf896445229935607fff166132d6dc0d26e02f6f7138d776b224a123b2193ff: Status 404 returned error can't find the container with id 8cf896445229935607fff166132d6dc0d26e02f6f7138d776b224a123b2193ff Nov 02 18:05:36.484013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.483980 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" event={"ID":"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6","Type":"ContainerStarted","Data":"43a5259a1f091a752a79dcd2b65de0aa80358ea05038f693d56cdac5f1f6300b"} Nov 02 18:05:36.484177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.484047 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" event={"ID":"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6","Type":"ContainerStarted","Data":"8cf896445229935607fff166132d6dc0d26e02f6f7138d776b224a123b2193ff"} Nov 02 18:05:36.485632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.485599 5647 generic.go:356] "Generic (PLEG): container finished" podID="793def51-6fec-4b5a-98d4-a4b0c33067a8" containerID="fca0242590a68cb816f7b1ab7a118263429ee8a32c51ddba302e716c5178a2f1" exitCode=0 Nov 02 18:05:36.485748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.485689 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" event={"ID":"793def51-6fec-4b5a-98d4-a4b0c33067a8","Type":"ContainerDied","Data":"fca0242590a68cb816f7b1ab7a118263429ee8a32c51ddba302e716c5178a2f1"} Nov 02 18:05:36.485809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.485757 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" event={"ID":"793def51-6fec-4b5a-98d4-a4b0c33067a8","Type":"ContainerStarted","Data":"c6d7a9c4bf64a88799c1a209c80a890e47e85985734239fc8b50f1fce2cc5ea3"} Nov 02 18:05:36.604819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.603102 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" podStartSLOduration=1.603085468 podStartE2EDuration="1.603085468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:36.602668173 +0000 UTC m=+2523.676195377" watchObservedRunningTime="2026-11-02 18:05:36.603085468 +0000 UTC m=+2523.676612672" Nov 02 18:05:36.697056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.697015 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q"] Nov 02 18:05:36.700972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.700950 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.758794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.758723 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.758826 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.758864 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9dj\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.758919 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.759022 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.759053 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.759136 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.759164 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.759408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.759205 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.792128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.792094 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q"] Nov 02 18:05:36.860316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860215 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860277 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860312 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9dj\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860376 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860435 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860465 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860522 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860552 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.860638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860610 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.861003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.860714 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.861128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.861099 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.861440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.861411 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.861440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.861430 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.861583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.861513 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.862700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.862677 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.863024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.863000 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.920827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.920797 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:36.921048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:36.921024 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9dj\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:37.015305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.015255 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:37.450355 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.450323 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q"] Nov 02 18:05:37.451869 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:37.451833 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2af564_d05a_412a_bea6_fc57f6b06638.slice/crio-1cab1637df3911ae326977e062e1f0d2a73399cf8d6e5c864e253b1b714aaaa4 WatchSource:0}: Error finding container 1cab1637df3911ae326977e062e1f0d2a73399cf8d6e5c864e253b1b714aaaa4: Status 404 returned error can't find the container with id 1cab1637df3911ae326977e062e1f0d2a73399cf8d6e5c864e253b1b714aaaa4 Nov 02 18:05:37.454779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.454727 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:05:37.454893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.454822 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:05:37.454893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.454852 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:05:37.494007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.493973 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" event={"ID":"bb2af564-d05a-412a-bea6-fc57f6b06638","Type":"ContainerStarted","Data":"1cab1637df3911ae326977e062e1f0d2a73399cf8d6e5c864e253b1b714aaaa4"} Nov 02 18:05:37.495702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.495675 5647 generic.go:356] "Generic (PLEG): container finished" podID="a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" containerID="43a5259a1f091a752a79dcd2b65de0aa80358ea05038f693d56cdac5f1f6300b" exitCode=0 Nov 02 18:05:37.495875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.495706 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" event={"ID":"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6","Type":"ContainerDied","Data":"43a5259a1f091a752a79dcd2b65de0aa80358ea05038f693d56cdac5f1f6300b"} Nov 02 18:05:37.659997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.659963 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:37.783464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.783376 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca\") pod \"793def51-6fec-4b5a-98d4-a4b0c33067a8\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " Nov 02 18:05:37.783464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.783453 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w\" (UniqueName: \"kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w\") pod \"793def51-6fec-4b5a-98d4-a4b0c33067a8\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " Nov 02 18:05:37.783725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.783544 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca\") pod \"793def51-6fec-4b5a-98d4-a4b0c33067a8\" (UID: \"793def51-6fec-4b5a-98d4-a4b0c33067a8\") " Nov 02 18:05:37.784240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.784209 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca" pod "793def51-6fec-4b5a-98d4-a4b0c33067a8" (UID: "793def51-6fec-4b5a-98d4-a4b0c33067a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:37.786331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.786299 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca" pod "793def51-6fec-4b5a-98d4-a4b0c33067a8" (UID: "793def51-6fec-4b5a-98d4-a4b0c33067a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:37.786899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.786865 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w" pod "793def51-6fec-4b5a-98d4-a4b0c33067a8" (UID: "793def51-6fec-4b5a-98d4-a4b0c33067a8"). InnerVolumeSpecName "kube-api-access-8pk8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:37.884765 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.884710 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:37.884765 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.884759 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pk8w\" (UniqueName: \"kubernetes.io/projected/793def51-6fec-4b5a-98d4-a4b0c33067a8-kube-api-access-8pk8w\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:37.884765 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:37.884769 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/793def51-6fec-4b5a-98d4-a4b0c33067a8-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:38.511822 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.503864 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" event={"ID":"bb2af564-d05a-412a-bea6-fc57f6b06638","Type":"ContainerStarted","Data":"b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282"} Nov 02 18:05:38.527528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.527485 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" event={"ID":"793def51-6fec-4b5a-98d4-a4b0c33067a8","Type":"ContainerDied","Data":"c6d7a9c4bf64a88799c1a209c80a890e47e85985734239fc8b50f1fce2cc5ea3"} Nov 02 18:05:38.527777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.527559 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d7a9c4bf64a88799c1a209c80a890e47e85985734239fc8b50f1fce2cc5ea3" Nov 02 18:05:38.529165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.527904 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2" Nov 02 18:05:38.733241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.733208 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:38.753474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.753421 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podStartSLOduration=2.753406564 podStartE2EDuration="2.753406564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:38.70150951 +0000 UTC m=+2525.775036716" watchObservedRunningTime="2026-11-02 18:05:38.753406564 +0000 UTC m=+2525.826933767" Nov 02 18:05:38.796549 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.796469 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584\" (UniqueName: \"kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584\") pod \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " Nov 02 18:05:38.796549 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.796521 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca\") pod \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " Nov 02 18:05:38.796769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.796697 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca\") pod \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\" (UID: \"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6\") " Nov 02 18:05:38.797042 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.797014 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca" pod "a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" (UID: "a0bdbc45-6cde-4466-b68a-4dcb6721c4d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:38.797303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.797287 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:38.798805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.798781 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca" pod "a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" (UID: "a0bdbc45-6cde-4466-b68a-4dcb6721c4d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:38.798953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.798823 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584" pod "a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" (UID: "a0bdbc45-6cde-4466-b68a-4dcb6721c4d6"). InnerVolumeSpecName "kube-api-access-td584". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:38.898788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.898745 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td584\" (UniqueName: \"kubernetes.io/projected/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-kube-api-access-td584\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:38.898788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:38.898783 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:39.026885 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:39.026846 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:39.027155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:39.026909 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 02 18:05:39.027287 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:39.027192 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 02 18:05:39.534821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:39.534713 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" event={"ID":"a0bdbc45-6cde-4466-b68a-4dcb6721c4d6","Type":"ContainerDied","Data":"8cf896445229935607fff166132d6dc0d26e02f6f7138d776b224a123b2193ff"} Nov 02 18:05:39.534821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:39.534750 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg" Nov 02 18:05:39.534821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:39.534777 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf896445229935607fff166132d6dc0d26e02f6f7138d776b224a123b2193ff" Nov 02 18:05:40.023760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:40.023240 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 02 18:05:40.023760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:40.023304 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 02 18:05:41.018166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:41.018124 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 02 18:05:41.018794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:41.018716 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 02 18:05:42.024761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:42.022238 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 02 18:05:42.024761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:42.022303 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 02 18:05:43.024760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:43.024203 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 02 18:05:43.024760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:43.024277 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 02 18:05:44.015674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:44.015635 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Nov 02 18:05:44.015883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:44.015705 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Nov 02 18:05:45.020782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:45.020712 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:45.021312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:45.021290 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:45.022081 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:45.022058 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:05:49.173003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.172967 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd"] Nov 02 18:05:49.173463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.173449 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" containerName="revoke-keys" Nov 02 18:05:49.173511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.173465 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" containerName="revoke-keys" Nov 02 18:05:49.173511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.173493 5647 cpu_manager.go:414] "containerMap: removing container" podUID="793def51-6fec-4b5a-98d4-a4b0c33067a8" containerName="revoke-keys" Nov 02 18:05:49.173511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.173498 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793def51-6fec-4b5a-98d4-a4b0c33067a8" containerName="revoke-keys" Nov 02 18:05:49.173612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.173575 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="793def51-6fec-4b5a-98d4-a4b0c33067a8" containerName="revoke-keys" Nov 02 18:05:49.173612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.173589 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" containerName="revoke-keys" Nov 02 18:05:49.177328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.177305 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.202462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.202419 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.202903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.202875 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.203181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.203125 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.203479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.203434 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.225866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.225828 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpnk2\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.227486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.227445 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.227656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.227527 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.227656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.227559 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.227656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.227586 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.227905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.227377 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865"] Nov 02 18:05:49.232761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.232603 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.250757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.250112 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-dd92ac-openshift-default-dockercfg-m2cfj\"" Nov 02 18:05:49.263756 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.258776 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd"] Nov 02 18:05:49.331906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.331867 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865"] Nov 02 18:05:49.332630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332602 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.332686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332660 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.332782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332760 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tpz\" (UniqueName: \"kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.332833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332802 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.332867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332842 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpnk2\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.332919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332878 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.332958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332932 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.332994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332967 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.333030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.332995 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.333069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.333027 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.333897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.333873 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.334087 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.334067 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.334170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.334154 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.335187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.335161 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.335440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.335420 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.336299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.336274 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.337083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.337057 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:05:49.338111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.338091 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:05:49.341094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.341071 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.406082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.398413 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.410083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.410052 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.410402 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.410376 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpnk2\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.416323 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.416273 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token\") pod \"e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.435557 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.435489 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.435696 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.435573 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.435696 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.435643 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tpz\" (UniqueName: \"kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.436173 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.436152 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.438098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.438073 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.491124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.491090 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:49.494101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.494069 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tpz\" (UniqueName: \"kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz\") pod \"maas-api-revoke-sub-cddd2ba8-5b865\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.724881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.724800 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:49.925510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.925487 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd"] Nov 02 18:05:49.926496 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:49.926472 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59bd3539_3194_4879_87b9_48a316cda7eb.slice/crio-768264854e5d47da61e03f3e25b6b88cfc5321aa8b92b096885dfb92e4070f2e WatchSource:0}: Error finding container 768264854e5d47da61e03f3e25b6b88cfc5321aa8b92b096885dfb92e4070f2e: Status 404 returned error can't find the container with id 768264854e5d47da61e03f3e25b6b88cfc5321aa8b92b096885dfb92e4070f2e Nov 02 18:05:49.928818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.928786 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:05:49.928924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.928850 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:05:49.928924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:49.928878 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:05:50.112469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.112447 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865"] Nov 02 18:05:50.114074 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:50.114047 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba20f5a_781e_4204_ac0c_3f02c509f918.slice/crio-42b026e6c916a7a93ca1381796910675074f168d50d86d384c2eac5c5daed7aa WatchSource:0}: Error finding container 42b026e6c916a7a93ca1381796910675074f168d50d86d384c2eac5c5daed7aa: Status 404 returned error can't find the container with id 42b026e6c916a7a93ca1381796910675074f168d50d86d384c2eac5c5daed7aa Nov 02 18:05:50.722530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.722498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" event={"ID":"4ba20f5a-781e-4204-ac0c-3f02c509f918","Type":"ContainerStarted","Data":"c577f0ef46d69add00b3c9bcd37629fcf39c340ae2f0157dd6cfb0cba0431c42"} Nov 02 18:05:50.723018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.722560 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" event={"ID":"4ba20f5a-781e-4204-ac0c-3f02c509f918","Type":"ContainerStarted","Data":"42b026e6c916a7a93ca1381796910675074f168d50d86d384c2eac5c5daed7aa"} Nov 02 18:05:50.724186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.724162 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" event={"ID":"59bd3539-3194-4879-87b9-48a316cda7eb","Type":"ContainerStarted","Data":"e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14"} Nov 02 18:05:50.724263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.724204 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" event={"ID":"59bd3539-3194-4879-87b9-48a316cda7eb","Type":"ContainerStarted","Data":"768264854e5d47da61e03f3e25b6b88cfc5321aa8b92b096885dfb92e4070f2e"} Nov 02 18:05:50.840096 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.840045 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" podStartSLOduration=1.840028723 podStartE2EDuration="1.840028723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:50.83849062 +0000 UTC m=+2537.912017824" watchObservedRunningTime="2026-11-02 18:05:50.840028723 +0000 UTC m=+2537.913555926" Nov 02 18:05:50.961389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:50.961336 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podStartSLOduration=1.961321595 podStartE2EDuration="1.961321595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:50.957257734 +0000 UTC m=+2538.030784937" watchObservedRunningTime="2026-11-02 18:05:50.961321595 +0000 UTC m=+2538.034848798" Nov 02 18:05:51.491294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:51.491257 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:51.496818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:51.496791 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:51.732084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:51.732043 5647 generic.go:356] "Generic (PLEG): container finished" podID="4ba20f5a-781e-4204-ac0c-3f02c509f918" containerID="c577f0ef46d69add00b3c9bcd37629fcf39c340ae2f0157dd6cfb0cba0431c42" exitCode=0 Nov 02 18:05:51.732602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:51.732135 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" event={"ID":"4ba20f5a-781e-4204-ac0c-3f02c509f918","Type":"ContainerDied","Data":"c577f0ef46d69add00b3c9bcd37629fcf39c340ae2f0157dd6cfb0cba0431c42"} Nov 02 18:05:51.732602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:51.732455 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:51.733386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:51.733363 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:05:52.880508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:52.880463 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:52.996770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:52.996654 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca\") pod \"4ba20f5a-781e-4204-ac0c-3f02c509f918\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " Nov 02 18:05:52.996770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:52.996714 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca\") pod \"4ba20f5a-781e-4204-ac0c-3f02c509f918\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " Nov 02 18:05:52.997020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:52.996778 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz\" (UniqueName: \"kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz\") pod \"4ba20f5a-781e-4204-ac0c-3f02c509f918\" (UID: \"4ba20f5a-781e-4204-ac0c-3f02c509f918\") " Nov 02 18:05:52.997119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:52.997079 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca" pod "4ba20f5a-781e-4204-ac0c-3f02c509f918" (UID: "4ba20f5a-781e-4204-ac0c-3f02c509f918"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:05:52.997550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:52.997365 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:53.000889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.000845 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz" pod "4ba20f5a-781e-4204-ac0c-3f02c509f918" (UID: "4ba20f5a-781e-4204-ac0c-3f02c509f918"). InnerVolumeSpecName "kube-api-access-b2tpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:05:53.001007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.000895 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca" pod "4ba20f5a-781e-4204-ac0c-3f02c509f918" (UID: "4ba20f5a-781e-4204-ac0c-3f02c509f918"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:05:53.098795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.098764 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ba20f5a-781e-4204-ac0c-3f02c509f918-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:53.098795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.098793 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2tpz\" (UniqueName: \"kubernetes.io/projected/4ba20f5a-781e-4204-ac0c-3f02c509f918-kube-api-access-b2tpz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:05:53.745209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.745179 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:05:53.745410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.745224 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" event={"ID":"4ba20f5a-781e-4204-ac0c-3f02c509f918","Type":"ContainerDied","Data":"42b026e6c916a7a93ca1381796910675074f168d50d86d384c2eac5c5daed7aa"} Nov 02 18:05:53.745410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:53.745281 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b026e6c916a7a93ca1381796910675074f168d50d86d384c2eac5c5daed7aa" Nov 02 18:05:57.945889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:57.945840 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x"] Nov 02 18:05:57.946662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:57.946637 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4ba20f5a-781e-4204-ac0c-3f02c509f918" containerName="revoke-keys" Nov 02 18:05:57.947009 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:57.946988 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba20f5a-781e-4204-ac0c-3f02c509f918" containerName="revoke-keys" Nov 02 18:05:57.947447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:57.947428 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba20f5a-781e-4204-ac0c-3f02c509f918" containerName="revoke-keys" Nov 02 18:05:57.980060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:57.980023 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:57.997160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:57.997111 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x"] Nov 02 18:05:58.105495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.105457 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz"] Nov 02 18:05:58.109300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.109276 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.149070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.149039 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-dd92ac\"" Nov 02 18:05:58.149244 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.149233 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-dd92ac-dockercfg-jsqbl\"" Nov 02 18:05:58.164514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.164467 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.165359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.165332 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkfh6\" (UniqueName: \"kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6\") pod \"payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.169598 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.169575 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz"] Nov 02 18:05:58.266546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.266465 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2mn\" (UniqueName: \"kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn\") pod \"payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.266546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.266538 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkfh6\" (UniqueName: \"kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6\") pod \"payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.266797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.266641 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.266797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.266765 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.267418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.267397 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.325876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.325839 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkfh6\" (UniqueName: \"kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6\") pod \"payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.368104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.368068 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2mn\" (UniqueName: \"kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn\") pod \"payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.368311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.368147 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.368897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.368874 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.427158 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.427123 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2mn\" (UniqueName: \"kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn\") pod \"payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.611820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.611789 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:58.722667 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.722634 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:58.757145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.757104 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5"] Nov 02 18:05:58.764259 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.764230 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.817277 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.817215 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm"] Nov 02 18:05:58.819970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.819534 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:05:58.822899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.822873 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:05:58.825499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.825435 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5"] Nov 02 18:05:58.825643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.825576 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:58.874614 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.874526 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.875277 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.874649 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lv5\" (UniqueName: \"kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.875474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.874842 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.879022 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.878981 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm"] Nov 02 18:05:58.976524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.976486 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lv5\" (UniqueName: \"kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.977036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.976557 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:58.977036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.976773 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.977036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.976795 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfm45\" (UniqueName: \"kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:58.977036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.976823 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:58.977036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.976961 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.977534 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.977515 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.979305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.979287 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:58.994755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:58.994713 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x"] Nov 02 18:05:58.996342 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:58.996315 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6e4cd1_24fe_400c_b01d_4f7b6a83e731.slice/crio-4ffe6017b3606d0b22e8e84dafca7afdb4817b7d85301867b2785a5cb35c02a2 WatchSource:0}: Error finding container 4ffe6017b3606d0b22e8e84dafca7afdb4817b7d85301867b2785a5cb35c02a2: Status 404 returned error can't find the container with id 4ffe6017b3606d0b22e8e84dafca7afdb4817b7d85301867b2785a5cb35c02a2 Nov 02 18:05:59.035610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.035587 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lv5\" (UniqueName: \"kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5\") pod \"maas-api-revoke-sub-4f5f1fcb-h4gf5\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:59.078497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.078464 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.078648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.078530 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfm45\" (UniqueName: \"kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.078648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.078574 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.079448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.079373 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.081286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.081264 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.087264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.087241 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:05:59.110404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.110375 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz"] Nov 02 18:05:59.112160 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:59.112127 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87d3f30_6f61_4eea_b2da_ea6b9ac665b5.slice/crio-87c95f415fd286b66dfacaac3ddbb9f7ee48e4b4c5ff45689219c762edd35489 WatchSource:0}: Error finding container 87c95f415fd286b66dfacaac3ddbb9f7ee48e4b4c5ff45689219c762edd35489: Status 404 returned error can't find the container with id 87c95f415fd286b66dfacaac3ddbb9f7ee48e4b4c5ff45689219c762edd35489 Nov 02 18:05:59.137973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.137944 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfm45\" (UniqueName: \"kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45\") pod \"maas-api-revoke-sub-87442f3b-svxtm\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.146510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.145913 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:05:59.470093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.470049 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5"] Nov 02 18:05:59.471084 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:59.471051 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b790370_dece_408f_9137_69f4b41fe345.slice/crio-a0e76c98d7912ea5444c5d908a1ad3284633f27ccc612df074faeebcfb41c5d7 WatchSource:0}: Error finding container a0e76c98d7912ea5444c5d908a1ad3284633f27ccc612df074faeebcfb41c5d7: Status 404 returned error can't find the container with id a0e76c98d7912ea5444c5d908a1ad3284633f27ccc612df074faeebcfb41c5d7 Nov 02 18:05:59.580693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.580665 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm"] Nov 02 18:05:59.581713 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:05:59.581689 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b3409e_72bc_4c58_a7fc_783dd9c8cf9d.slice/crio-e98241f431a9a0edae077bb14701d919978deba778d1486385e0ea4b7fb6524b WatchSource:0}: Error finding container e98241f431a9a0edae077bb14701d919978deba778d1486385e0ea4b7fb6524b: Status 404 returned error can't find the container with id e98241f431a9a0edae077bb14701d919978deba778d1486385e0ea4b7fb6524b Nov 02 18:05:59.809353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.809317 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" event={"ID":"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731","Type":"ContainerStarted","Data":"88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8"} Nov 02 18:05:59.809547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.809514 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:05:59.809623 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.809558 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" event={"ID":"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731","Type":"ContainerStarted","Data":"4ffe6017b3606d0b22e8e84dafca7afdb4817b7d85301867b2785a5cb35c02a2"} Nov 02 18:05:59.811225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.811199 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" event={"ID":"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5","Type":"ContainerStarted","Data":"a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a"} Nov 02 18:05:59.811356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.811251 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:05:59.811356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.811276 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" event={"ID":"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5","Type":"ContainerStarted","Data":"87c95f415fd286b66dfacaac3ddbb9f7ee48e4b4c5ff45689219c762edd35489"} Nov 02 18:05:59.812622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.812598 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" event={"ID":"2b790370-dece-408f-9137-69f4b41fe345","Type":"ContainerStarted","Data":"a0e76c98d7912ea5444c5d908a1ad3284633f27ccc612df074faeebcfb41c5d7"} Nov 02 18:05:59.813787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.813761 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" event={"ID":"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d","Type":"ContainerStarted","Data":"e98241f431a9a0edae077bb14701d919978deba778d1486385e0ea4b7fb6524b"} Nov 02 18:05:59.927252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:05:59.927203 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" podStartSLOduration=2.927190552 podStartE2EDuration="2.927190552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:59.926304977 +0000 UTC m=+2546.999832180" watchObservedRunningTime="2026-11-02 18:05:59.927190552 +0000 UTC m=+2547.000717754" Nov 02 18:06:00.046682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:00.046628 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" podStartSLOduration=2.04661333 podStartE2EDuration="2.04661333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:00.045636099 +0000 UTC m=+2547.119163306" watchObservedRunningTime="2026-11-02 18:06:00.04661333 +0000 UTC m=+2547.120140601" Nov 02 18:06:00.821120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:00.821031 5647 generic.go:356] "Generic (PLEG): container finished" podID="2b790370-dece-408f-9137-69f4b41fe345" containerID="3ce4bcb2a2a224db897d450b176bc39ae8bc87cc9fe1e46fafce081c648f96cf" exitCode=0 Nov 02 18:06:00.821278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:00.821116 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" event={"ID":"2b790370-dece-408f-9137-69f4b41fe345","Type":"ContainerDied","Data":"3ce4bcb2a2a224db897d450b176bc39ae8bc87cc9fe1e46fafce081c648f96cf"} Nov 02 18:06:00.822793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:00.822763 5647 generic.go:356] "Generic (PLEG): container finished" podID="77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" containerID="6fc3570136c3d9db8ed39e5cfd21444ea982919bb0a24c7c431171344dcb06d6" exitCode=0 Nov 02 18:06:00.822957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:00.822847 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" event={"ID":"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d","Type":"ContainerDied","Data":"6fc3570136c3d9db8ed39e5cfd21444ea982919bb0a24c7c431171344dcb06d6"} Nov 02 18:06:01.977956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:01.977918 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:06:02.013668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.013643 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:06:02.114807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.114766 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45\" (UniqueName: \"kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45\") pod \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " Nov 02 18:06:02.114973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.114836 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca\") pod \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " Nov 02 18:06:02.114973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.114871 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca\") pod \"2b790370-dece-408f-9137-69f4b41fe345\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " Nov 02 18:06:02.114973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.114918 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca\") pod \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\" (UID: \"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d\") " Nov 02 18:06:02.115137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.115003 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca\") pod \"2b790370-dece-408f-9137-69f4b41fe345\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " Nov 02 18:06:02.115137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.115082 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5\" (UniqueName: \"kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5\") pod \"2b790370-dece-408f-9137-69f4b41fe345\" (UID: \"2b790370-dece-408f-9137-69f4b41fe345\") " Nov 02 18:06:02.115328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.115298 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca" pod "2b790370-dece-408f-9137-69f4b41fe345" (UID: "2b790370-dece-408f-9137-69f4b41fe345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:02.115424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.115355 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca" pod "77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" (UID: "77b3409e-72bc-4c58-a7fc-783dd9c8cf9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:02.115877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.115847 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b790370-dece-408f-9137-69f4b41fe345-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:02.115877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.115875 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:02.117362 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.117333 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca" pod "2b790370-dece-408f-9137-69f4b41fe345" (UID: "2b790370-dece-408f-9137-69f4b41fe345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:02.117362 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.117341 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca" pod "77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" (UID: "77b3409e-72bc-4c58-a7fc-783dd9c8cf9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:02.117693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.117671 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45" pod "77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" (UID: "77b3409e-72bc-4c58-a7fc-783dd9c8cf9d"). InnerVolumeSpecName "kube-api-access-bfm45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:02.117803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.117785 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5" pod "2b790370-dece-408f-9137-69f4b41fe345" (UID: "2b790370-dece-408f-9137-69f4b41fe345"). InnerVolumeSpecName "kube-api-access-q7lv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:02.219048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.216976 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lv5\" (UniqueName: \"kubernetes.io/projected/2b790370-dece-408f-9137-69f4b41fe345-kube-api-access-q7lv5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:02.219048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.217008 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfm45\" (UniqueName: \"kubernetes.io/projected/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-kube-api-access-bfm45\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:02.219048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.217022 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:02.219048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.217034 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b790370-dece-408f-9137-69f4b41fe345-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:02.835718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.835685 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:06:02.835908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.835714 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" event={"ID":"77b3409e-72bc-4c58-a7fc-783dd9c8cf9d","Type":"ContainerDied","Data":"e98241f431a9a0edae077bb14701d919978deba778d1486385e0ea4b7fb6524b"} Nov 02 18:06:02.835908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.835806 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98241f431a9a0edae077bb14701d919978deba778d1486385e0ea4b7fb6524b" Nov 02 18:06:02.837325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.837300 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" event={"ID":"2b790370-dece-408f-9137-69f4b41fe345","Type":"ContainerDied","Data":"a0e76c98d7912ea5444c5d908a1ad3284633f27ccc612df074faeebcfb41c5d7"} Nov 02 18:06:02.837325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.837315 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5" Nov 02 18:06:02.837538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:02.837338 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e76c98d7912ea5444c5d908a1ad3284633f27ccc612df074faeebcfb41c5d7" Nov 02 18:06:03.203571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.203536 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm"] Nov 02 18:06:03.204102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.204086 5647 cpu_manager.go:414] "containerMap: removing container" podUID="77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" containerName="revoke-keys" Nov 02 18:06:03.204156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.204103 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" containerName="revoke-keys" Nov 02 18:06:03.204156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.204124 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2b790370-dece-408f-9137-69f4b41fe345" containerName="revoke-keys" Nov 02 18:06:03.204156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.204130 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b790370-dece-408f-9137-69f4b41fe345" containerName="revoke-keys" Nov 02 18:06:03.204257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.204236 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b790370-dece-408f-9137-69f4b41fe345" containerName="revoke-keys" Nov 02 18:06:03.204257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.204253 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" containerName="revoke-keys" Nov 02 18:06:03.207831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.207809 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.223944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.223911 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg"] Nov 02 18:06:03.227963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.227943 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231099 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231078 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231113 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231142 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231268 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231317 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8msp\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231347 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231426 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231498 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231535 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231560 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231591 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231622 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.231835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231778 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231878 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231909 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.231985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231968 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.232041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.231995 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfrb\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.232120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.232097 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.271802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.271757 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6e3175-openshift-default-dockercfg-n7khr\"" Nov 02 18:06:03.307108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.307072 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm"] Nov 02 18:06:03.333532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333498 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.333755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333540 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.333755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333613 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.333755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333640 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.333755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333666 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.333755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333689 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfrb\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.333755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333749 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333856 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333890 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333925 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333954 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.333984 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8msp\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334013 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334044 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334079 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334105 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334151 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334176 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334213 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334338 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334381 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334512 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.334608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334525 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.335045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334787 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.335045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.334895 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.335171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.335079 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.335230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.335199 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.335291 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.335224 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.337714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.337681 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.337921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.337778 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.337921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.337915 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.338177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.338155 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.356325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.356295 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg"] Nov 02 18:06:03.394216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.394182 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.394498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.394475 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8msp\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.394561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.394547 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.394608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.394562 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfrb\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.522522 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.522423 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:03.541814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.541790 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:03.965455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.965406 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg"] Nov 02 18:06:03.967191 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:03.967108 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa77dbd2_aec7_4e43_93ee_61772d5e6953.slice/crio-85c3f9c1623bbeff44269391cda29824d5e7d12030ffd39920dfe6a0f29cf1f9 WatchSource:0}: Error finding container 85c3f9c1623bbeff44269391cda29824d5e7d12030ffd39920dfe6a0f29cf1f9: Status 404 returned error can't find the container with id 85c3f9c1623bbeff44269391cda29824d5e7d12030ffd39920dfe6a0f29cf1f9 Nov 02 18:06:03.970273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.970243 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm"] Nov 02 18:06:03.970418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.970356 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:03.970500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.970446 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:03.970500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.970491 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:03.970778 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:03.970714 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080bf38f_c034_4ccd_8c09_e004c25578e5.slice/crio-31adc099e22604da3a9b865415ed54c97011d66f8b92d150db64760004141a10 WatchSource:0}: Error finding container 31adc099e22604da3a9b865415ed54c97011d66f8b92d150db64760004141a10: Status 404 returned error can't find the container with id 31adc099e22604da3a9b865415ed54c97011d66f8b92d150db64760004141a10 Nov 02 18:06:03.973393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.973360 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:03.973463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.973445 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:03.973514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:03.973499 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:04.851428 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:04.851388 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" event={"ID":"fa77dbd2-aec7-4e43-93ee-61772d5e6953","Type":"ContainerStarted","Data":"799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528"} Nov 02 18:06:04.852025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:04.851444 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" event={"ID":"fa77dbd2-aec7-4e43-93ee-61772d5e6953","Type":"ContainerStarted","Data":"85c3f9c1623bbeff44269391cda29824d5e7d12030ffd39920dfe6a0f29cf1f9"} Nov 02 18:06:04.853167 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:04.853142 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" event={"ID":"080bf38f-c034-4ccd-8c09-e004c25578e5","Type":"ContainerStarted","Data":"cd08f2bd63af1bf8434fa22552d02c76da42891e943852100fc5428c328d462b"} Nov 02 18:06:04.853285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:04.853196 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" event={"ID":"080bf38f-c034-4ccd-8c09-e004c25578e5","Type":"ContainerStarted","Data":"31adc099e22604da3a9b865415ed54c97011d66f8b92d150db64760004141a10"} Nov 02 18:06:05.201284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.201155 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podStartSLOduration=2.2011337380000002 podStartE2EDuration="2.201133738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:05.20090276 +0000 UTC m=+2552.274429963" watchObservedRunningTime="2026-11-02 18:06:05.201133738 +0000 UTC m=+2552.274660944" Nov 02 18:06:05.220761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.206357 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podStartSLOduration=2.206341781 podStartE2EDuration="2.206341781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:05.078060853 +0000 UTC m=+2552.151588056" watchObservedRunningTime="2026-11-02 18:06:05.206341781 +0000 UTC m=+2552.279868986" Nov 02 18:06:05.533821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.526147 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:05.551760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.549958 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:05.551760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.550038 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:05.551760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.550576 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:05.552250 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.552225 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:05.552426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:05.552402 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:05.977252 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:06:05.977213 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:06:06.536773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:06.535889 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:06.536773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:06.535965 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:06.551793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:06.546157 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:06.551793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:06.546220 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:07.523276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:07.523233 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:07.523971 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:07.523936 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:07.542485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:07.542349 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:07.542485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:07.542418 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:08.358076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.358033 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j"] Nov 02 18:06:08.363046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.363025 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.394989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.394953 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m689v\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.394999 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395082 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395134 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395170 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395202 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395248 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395267 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.395403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.395315 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.472393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.472358 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j"] Nov 02 18:06:08.497023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.496990 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497318 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497347 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497476 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m689v\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497517 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497562 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497616 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497667 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.497810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497792 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.498037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497853 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.498037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.497865 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.498154 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.498039 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.498530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.498504 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.500200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.500175 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.500816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.500795 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.557976 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.557938 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:08.559282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.558882 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:08.559282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.558354 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:08.559282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.558949 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:08.560794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.560763 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.561895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.561871 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m689v\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v\") pod \"e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:08.676439 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:08.676344 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:09.167033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.167003 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j"] Nov 02 18:06:09.168464 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:09.168409 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8ec92c_b980_4ac6_b145_123e091d1e5d.slice/crio-d41e1cd54938355378c961125d9eea55b4493d73c2aa86af838b0bccdb512eb8 WatchSource:0}: Error finding container d41e1cd54938355378c961125d9eea55b4493d73c2aa86af838b0bccdb512eb8: Status 404 returned error can't find the container with id d41e1cd54938355378c961125d9eea55b4493d73c2aa86af838b0bccdb512eb8 Nov 02 18:06:09.171029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.170993 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:09.171127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.171065 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:09.171127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.171094 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:09.525290 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.525195 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:09.525625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.525592 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:09.544160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.544121 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:09.544436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.544399 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:09.568178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.568140 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl"] Nov 02 18:06:09.573013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.572987 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.633759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.633040 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:06:09.633759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.633247 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:06:09.650761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.641583 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2vrj\" (UniqueName: \"kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.650761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.641679 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.650761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.641748 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.663821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.659300 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl"] Nov 02 18:06:09.763860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.753646 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:06:09.763860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.755188 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.763860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.755239 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.763860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.755399 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2vrj\" (UniqueName: \"kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.765412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.765376 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.769172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.769070 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.839120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.825877 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2vrj\" (UniqueName: \"kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj\") pod \"maas-api-revoke-sub-64a8c37c-cb9sl\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.904776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.891970 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:09.982692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.982355 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" event={"ID":"9f8ec92c-b980-4ac6-b145-123e091d1e5d","Type":"ContainerStarted","Data":"7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886"} Nov 02 18:06:09.982692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:09.982407 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" event={"ID":"9f8ec92c-b980-4ac6-b145-123e091d1e5d","Type":"ContainerStarted","Data":"d41e1cd54938355378c961125d9eea55b4493d73c2aa86af838b0bccdb512eb8"} Nov 02 18:06:10.564620 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.537516 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:10.564620 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.537593 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:10.601960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.576392 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:10.601960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.576462 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:10.733801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.730330 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podStartSLOduration=2.7303060390000002 podStartE2EDuration="2.730306039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:10.213509053 +0000 UTC m=+2557.287036270" watchObservedRunningTime="2026-11-02 18:06:10.730306039 +0000 UTC m=+2557.803833244" Nov 02 18:06:10.733801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.733470 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:10.733801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.733541 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl"] Nov 02 18:06:10.737323 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.736114 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:10.737323 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:10.736178 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:11.326783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.323871 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:11.370373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.370329 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" event={"ID":"2a187343-1ee4-4bea-8672-f35027e1b89d","Type":"ContainerStarted","Data":"686dd724dc8f9f22ce093fc4aad483b7344111f0afe3ea9727f31049c4b976e3"} Nov 02 18:06:11.372530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.372500 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp"] Nov 02 18:06:11.386013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.385980 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:06:11.395831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.386297 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.396479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.396004 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp"] Nov 02 18:06:11.396807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.396788 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:06:11.474460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.474391 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-580a21-openshift-default-dockercfg-zr6rg\"" Nov 02 18:06:11.483247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483085 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483373 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483412 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483520 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483565 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483652 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483682 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.483807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.483714 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.488177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.488147 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5c4f\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.532008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.531487 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Nov 02 18:06:11.532008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.531542 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Nov 02 18:06:11.555250 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.555125 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:11.555250 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.555182 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.614970 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615117 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615151 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615197 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615234 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615286 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615312 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615343 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.615383 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5c4f\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.638004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.632851 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.649618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.649180 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.678088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.663405 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.678088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.663660 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.678088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.663929 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.702787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.696184 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:11.702787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.696244 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:11.735237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.729359 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.801819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.767872 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.856030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.851508 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5c4f\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:11.856030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:11.852468 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy\") pod \"e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:12.033320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:12.033286 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:12.386773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:12.384380 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" event={"ID":"2a187343-1ee4-4bea-8672-f35027e1b89d","Type":"ContainerStarted","Data":"ad1bc39271de11e75f816715b908c0e3b96edf9e73455f1b5c6ba47cdf721051"} Nov 02 18:06:12.517911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:12.513696 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" podStartSLOduration=3.513674235 podStartE2EDuration="3.513674235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:12.510189903 +0000 UTC m=+2559.583717110" watchObservedRunningTime="2026-11-02 18:06:12.513674235 +0000 UTC m=+2559.587201441" Nov 02 18:06:12.588438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:12.546090 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Nov 02 18:06:12.588438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:12.546146 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Nov 02 18:06:13.003813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.000678 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:13.003813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.000770 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:13.195858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.195811 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm"] Nov 02 18:06:13.221958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.221917 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp"] Nov 02 18:06:13.222485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.222400 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.279786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.275828 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.279786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.275965 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdjmx\" (UniqueName: \"kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.279786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.276031 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.282936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.282577 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:13.282936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.282665 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:13.282936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.282710 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:13.282936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.282896 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm"] Nov 02 18:06:13.389761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.377184 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.393756 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.390179 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdjmx\" (UniqueName: \"kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.413762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.397864 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.451760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.441369 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.492958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.491214 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:13.498512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.498092 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.523795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.503249 5647 generic.go:356] "Generic (PLEG): container finished" podID="2a187343-1ee4-4bea-8672-f35027e1b89d" containerID="ad1bc39271de11e75f816715b908c0e3b96edf9e73455f1b5c6ba47cdf721051" exitCode=0 Nov 02 18:06:13.523795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.519002 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" event={"ID":"2a187343-1ee4-4bea-8672-f35027e1b89d","Type":"ContainerDied","Data":"ad1bc39271de11e75f816715b908c0e3b96edf9e73455f1b5c6ba47cdf721051"} Nov 02 18:06:13.523795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.522679 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdjmx\" (UniqueName: \"kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx\") pod \"maas-api-revoke-sub-29da9c56-t6fbm\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.533861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.531834 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:06:13.533861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.531927 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:13.563762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.559926 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" event={"ID":"54da1589-034c-48a5-b9dc-b10a59c90b76","Type":"ContainerStarted","Data":"8bf18757013ea3428813c55c5927bba684ac02835c11bfc0f3930ff48391a680"} Nov 02 18:06:13.594399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.594351 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:13.714766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.713304 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:13.714766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:13.713378 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:14.107147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.100938 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm"] Nov 02 18:06:14.121372 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:14.121308 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f79ccba_dfd1_4b0c_9267_9c997de03c00.slice/crio-45827da937fc3cc98fdebeba0cbbebfca14b74bb018cf252423d1bc0150b0ed1 WatchSource:0}: Error finding container 45827da937fc3cc98fdebeba0cbbebfca14b74bb018cf252423d1bc0150b0ed1: Status 404 returned error can't find the container with id 45827da937fc3cc98fdebeba0cbbebfca14b74bb018cf252423d1bc0150b0ed1 Nov 02 18:06:14.550814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.550715 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:06:14.551574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.551105 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:14.592145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.592102 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:06:14.592348 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.592180 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:14.595461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.595340 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" event={"ID":"1f79ccba-dfd1-4b0c-9267-9c997de03c00","Type":"ContainerStarted","Data":"45827da937fc3cc98fdebeba0cbbebfca14b74bb018cf252423d1bc0150b0ed1"} Nov 02 18:06:14.598969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.598889 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" event={"ID":"54da1589-034c-48a5-b9dc-b10a59c90b76","Type":"ContainerStarted","Data":"466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5"} Nov 02 18:06:14.687873 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.683956 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:14.691958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:14.691895 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:15.082773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.078258 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:15.097459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.097426 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:15.274236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.273935 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podStartSLOduration=5.273913202 podStartE2EDuration="5.273913202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:14.785153277 +0000 UTC m=+2561.858680483" watchObservedRunningTime="2026-11-02 18:06:15.273913202 +0000 UTC m=+2562.347440405" Nov 02 18:06:15.278007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.277983 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:15.348061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.348022 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:15.349821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.349798 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:15.351632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.351584 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:06:15.448681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.448627 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca\") pod \"2a187343-1ee4-4bea-8672-f35027e1b89d\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " Nov 02 18:06:15.449068 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.449041 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca\") pod \"2a187343-1ee4-4bea-8672-f35027e1b89d\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " Nov 02 18:06:15.449289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.449248 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca" pod "2a187343-1ee4-4bea-8672-f35027e1b89d" (UID: "2a187343-1ee4-4bea-8672-f35027e1b89d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:15.449368 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.449257 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj\" (UniqueName: \"kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj\") pod \"2a187343-1ee4-4bea-8672-f35027e1b89d\" (UID: \"2a187343-1ee4-4bea-8672-f35027e1b89d\") " Nov 02 18:06:15.450716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.450685 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:15.454209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.454173 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca" pod "2a187343-1ee4-4bea-8672-f35027e1b89d" (UID: "2a187343-1ee4-4bea-8672-f35027e1b89d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:15.456239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.456210 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj" pod "2a187343-1ee4-4bea-8672-f35027e1b89d" (UID: "2a187343-1ee4-4bea-8672-f35027e1b89d"). InnerVolumeSpecName "kube-api-access-v2vrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:15.560814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.557959 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a187343-1ee4-4bea-8672-f35027e1b89d-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:15.560814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.558004 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2vrj\" (UniqueName: \"kubernetes.io/projected/2a187343-1ee4-4bea-8672-f35027e1b89d-kube-api-access-v2vrj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:15.569392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.569356 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:06:15.569858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.569801 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:15.652946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.652912 5647 generic.go:356] "Generic (PLEG): container finished" podID="1f79ccba-dfd1-4b0c-9267-9c997de03c00" containerID="dc1bfd5ed8537e5b5edcf998ce370827dece9b94dbc0a1c8550a52f951550f98" exitCode=0 Nov 02 18:06:15.653156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.653132 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" event={"ID":"1f79ccba-dfd1-4b0c-9267-9c997de03c00","Type":"ContainerDied","Data":"dc1bfd5ed8537e5b5edcf998ce370827dece9b94dbc0a1c8550a52f951550f98"} Nov 02 18:06:15.654607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.654586 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" Nov 02 18:06:15.654793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.654774 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl" event={"ID":"2a187343-1ee4-4bea-8672-f35027e1b89d","Type":"ContainerDied","Data":"686dd724dc8f9f22ce093fc4aad483b7344111f0afe3ea9727f31049c4b976e3"} Nov 02 18:06:15.654947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.654933 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="686dd724dc8f9f22ce093fc4aad483b7344111f0afe3ea9727f31049c4b976e3" Nov 02 18:06:15.655341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.655325 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:15.665576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.664054 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:06:15.665576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.664280 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:15.666564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.665984 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:15.667931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.667901 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:06:15.677574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.677539 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:15.677718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:15.677594 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:16.677198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.677165 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 02 18:06:16.677595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.677228 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 02 18:06:16.832006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.831971 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:16.879512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.879468 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx\" (UniqueName: \"kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx\") pod \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " Nov 02 18:06:16.879766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.879653 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca\") pod \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " Nov 02 18:06:16.879766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.879691 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca\") pod \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\" (UID: \"1f79ccba-dfd1-4b0c-9267-9c997de03c00\") " Nov 02 18:06:16.880100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.880074 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca" pod "1f79ccba-dfd1-4b0c-9267-9c997de03c00" (UID: "1f79ccba-dfd1-4b0c-9267-9c997de03c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:06:16.880229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.880212 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:16.881861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.881835 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx" pod "1f79ccba-dfd1-4b0c-9267-9c997de03c00" (UID: "1f79ccba-dfd1-4b0c-9267-9c997de03c00"). InnerVolumeSpecName "kube-api-access-gdjmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:06:16.881987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.881947 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca" pod "1f79ccba-dfd1-4b0c-9267-9c997de03c00" (UID: "1f79ccba-dfd1-4b0c-9267-9c997de03c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:06:16.981338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.981231 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f79ccba-dfd1-4b0c-9267-9c997de03c00-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:16.981338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:16.981270 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdjmx\" (UniqueName: \"kubernetes.io/projected/1f79ccba-dfd1-4b0c-9267-9c997de03c00-kube-api-access-gdjmx\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:06:17.675161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:17.675129 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" Nov 02 18:06:17.676177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:17.676139 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm" event={"ID":"1f79ccba-dfd1-4b0c-9267-9c997de03c00","Type":"ContainerDied","Data":"45827da937fc3cc98fdebeba0cbbebfca14b74bb018cf252423d1bc0150b0ed1"} Nov 02 18:06:17.676395 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:17.676229 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45827da937fc3cc98fdebeba0cbbebfca14b74bb018cf252423d1bc0150b0ed1" Nov 02 18:06:17.846933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:17.846900 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:17.847350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:17.847186 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:17.848095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:17.848075 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:06:21.246471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.246429 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl"] Nov 02 18:06:21.247833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.247801 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1f79ccba-dfd1-4b0c-9267-9c997de03c00" containerName="revoke-keys" Nov 02 18:06:21.247833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.247837 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f79ccba-dfd1-4b0c-9267-9c997de03c00" containerName="revoke-keys" Nov 02 18:06:21.248071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.247893 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2a187343-1ee4-4bea-8672-f35027e1b89d" containerName="revoke-keys" Nov 02 18:06:21.248071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.247901 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a187343-1ee4-4bea-8672-f35027e1b89d" containerName="revoke-keys" Nov 02 18:06:21.248281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.248262 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f79ccba-dfd1-4b0c-9267-9c997de03c00" containerName="revoke-keys" Nov 02 18:06:21.248344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.248287 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a187343-1ee4-4bea-8672-f35027e1b89d" containerName="revoke-keys" Nov 02 18:06:21.253482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.253452 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.308205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.308171 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl"] Nov 02 18:06:21.344877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.344838 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.345079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.344935 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzj4v\" (UniqueName: \"kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v\") pod \"payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.367801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.365086 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-580a21-dockercfg-c7trx\"" Nov 02 18:06:21.368093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.367305 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-580a21\"" Nov 02 18:06:21.437758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.433948 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff"] Nov 02 18:06:21.446622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.446583 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.472762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.471430 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzj4v\" (UniqueName: \"kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v\") pod \"payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.472762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.471140 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.472762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.471042 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff"] Nov 02 18:06:21.472762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.471330 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.530936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.530900 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzj4v\" (UniqueName: \"kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v\") pod \"payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.569041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.569012 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:21.572467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.572441 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.572578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.572561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzsl\" (UniqueName: \"kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl\") pod \"payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.674066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.674031 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.674300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.674116 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzsl\" (UniqueName: \"kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl\") pod \"payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.674861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.674835 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.732579 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.732545 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzsl\" (UniqueName: \"kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl\") pod \"payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.807212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.807103 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:21.950762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:21.950714 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl"] Nov 02 18:06:21.951161 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:21.951127 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1d380e_5259_42c2_b9b9_ddc39ec1a280.slice/crio-aa299532f06d09c0da6523da7e0cf9cffb2ca1deb392e05a47b391ca9502e1ea WatchSource:0}: Error finding container aa299532f06d09c0da6523da7e0cf9cffb2ca1deb392e05a47b391ca9502e1ea: Status 404 returned error can't find the container with id aa299532f06d09c0da6523da7e0cf9cffb2ca1deb392e05a47b391ca9502e1ea Nov 02 18:06:22.034017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.033977 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47"] Nov 02 18:06:22.038037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.038018 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.137764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.137712 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47"] Nov 02 18:06:22.180612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180577 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.180800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180629 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.180800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180766 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.180800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180788 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.180952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180805 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.180952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180827 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9rb\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.180952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180882 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.181077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.180968 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.181077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.181008 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.187785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.187752 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff"] Nov 02 18:06:22.282357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282325 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282360 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282382 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282410 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9rb\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282444 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282515 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282565 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282702 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282767 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282785 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.282999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.282814 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.283664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.283640 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.283793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.283771 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.283919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.283893 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.285184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.285157 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.285858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.285838 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.343306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.343262 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9rb\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.343945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.343920 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token\") pod \"e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.351963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.351942 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:22.746412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.746314 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" event={"ID":"5b1d380e-5259-42c2-b9b9-ddc39ec1a280","Type":"ContainerStarted","Data":"8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64"} Nov 02 18:06:22.746412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.746381 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" event={"ID":"5b1d380e-5259-42c2-b9b9-ddc39ec1a280","Type":"ContainerStarted","Data":"aa299532f06d09c0da6523da7e0cf9cffb2ca1deb392e05a47b391ca9502e1ea"} Nov 02 18:06:22.746412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.746409 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:22.748246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.748222 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" event={"ID":"2eee8e39-18f1-443b-b577-b1f991f1e7c6","Type":"ContainerStarted","Data":"345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2"} Nov 02 18:06:22.748376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.748260 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" event={"ID":"2eee8e39-18f1-443b-b577-b1f991f1e7c6","Type":"ContainerStarted","Data":"827020a664521195e2116fb669a4fab998ad2171a776b4129b7fcce7f6108151"} Nov 02 18:06:22.748376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.748280 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:22.784046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.784023 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47"] Nov 02 18:06:22.785805 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:22.785778 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf814ce19_52ea_4bcc_ba18_ac5bd4b2e0c9.slice/crio-fb258122168dc968b17c26065c5568bc36d34d9fa813fa1c9395153252af0c46 WatchSource:0}: Error finding container fb258122168dc968b17c26065c5568bc36d34d9fa813fa1c9395153252af0c46: Status 404 returned error can't find the container with id fb258122168dc968b17c26065c5568bc36d34d9fa813fa1c9395153252af0c46 Nov 02 18:06:22.788469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.788431 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:22.788563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.788503 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:22.788563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.788552 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:22.873523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:22.873460 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" podStartSLOduration=1.8734391929999998 podStartE2EDuration="1.873439193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:22.868780589 +0000 UTC m=+2569.942307788" watchObservedRunningTime="2026-11-02 18:06:22.873439193 +0000 UTC m=+2569.946966397" Nov 02 18:06:23.012817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:23.007018 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" podStartSLOduration=2.006995079 podStartE2EDuration="2.006995079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:22.993133378 +0000 UTC m=+2570.066660587" watchObservedRunningTime="2026-11-02 18:06:23.006995079 +0000 UTC m=+2570.080522283" Nov 02 18:06:23.878151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:23.878113 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" event={"ID":"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9","Type":"ContainerStarted","Data":"e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487"} Nov 02 18:06:23.878559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:23.878207 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" event={"ID":"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9","Type":"ContainerStarted","Data":"fb258122168dc968b17c26065c5568bc36d34d9fa813fa1c9395153252af0c46"} Nov 02 18:06:24.013829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.006825 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podStartSLOduration=3.006804083 podStartE2EDuration="3.006804083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:24.006117044 +0000 UTC m=+2571.079644259" watchObservedRunningTime="2026-11-02 18:06:24.006804083 +0000 UTC m=+2571.080331287" Nov 02 18:06:24.069200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.069160 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm"] Nov 02 18:06:24.082567 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.082533 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm"] Nov 02 18:06:24.082785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.082664 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.190208 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.187307 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f7aa2e-openshift-default-dockercfg-s9k69\"" Nov 02 18:06:24.250841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.237526 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.251961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.251934 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.252335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.252307 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.252586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.252566 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.252706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.252688 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.252833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.252760 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.252833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.252815 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.252999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.252983 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.253079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.253051 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44hn\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.353031 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355503 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355582 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44hn\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355644 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355684 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355786 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355844 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355870 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.355917 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.356007 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.357284 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.358034 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.358645 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.358757 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.358804 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.359222 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.361767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.359475 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.372126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.372084 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.372126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.372082 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.430665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.430629 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.430665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.430651 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44hn\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn\") pod \"e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:24.699477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:24.699443 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:25.333820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.333791 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm"] Nov 02 18:06:25.335236 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:25.335203 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e44a8e2_8ff6_466d_a172_dd06de373029.slice/crio-5f0e666d3723d01e23fe746e63c9dfbf0e1d1bc76ab58cd44574d66ed1186d15 WatchSource:0}: Error finding container 5f0e666d3723d01e23fe746e63c9dfbf0e1d1bc76ab58cd44574d66ed1186d15: Status 404 returned error can't find the container with id 5f0e666d3723d01e23fe746e63c9dfbf0e1d1bc76ab58cd44574d66ed1186d15 Nov 02 18:06:25.337946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.337912 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:25.338040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.337980 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:25.338040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.338017 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:25.352559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.352531 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 02 18:06:25.352681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.352589 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 02 18:06:25.385374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.385343 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d"] Nov 02 18:06:25.392284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.392257 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.404840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.402465 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d"] Nov 02 18:06:25.447212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.447181 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-413a4e-openshift-default-dockercfg-dpc8p\"" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478559 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d94cg\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478660 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478693 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478773 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478827 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478868 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478965 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.478993 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.484784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.479083 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580177 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d94cg\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580233 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580255 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580292 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580328 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580350 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580422 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.580471 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.581591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.581089 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.582335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.582066 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.582335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.582304 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.582726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.582701 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.582972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.582951 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.584260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.584233 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.586422 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.586399 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.642541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.642503 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d94cg\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.648055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.647947 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token\") pod \"e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.741165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.741124 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:25.969047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.969009 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" event={"ID":"6e44a8e2-8ff6-466d-a172-dd06de373029","Type":"ContainerStarted","Data":"92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1"} Nov 02 18:06:25.969228 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:25.969066 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" event={"ID":"6e44a8e2-8ff6-466d-a172-dd06de373029","Type":"ContainerStarted","Data":"5f0e666d3723d01e23fe746e63c9dfbf0e1d1bc76ab58cd44574d66ed1186d15"} Nov 02 18:06:26.100279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.100227 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podStartSLOduration=3.100214109 podStartE2EDuration="3.100214109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:26.098506434 +0000 UTC m=+2573.172033637" watchObservedRunningTime="2026-11-02 18:06:26.100214109 +0000 UTC m=+2573.173741385" Nov 02 18:06:26.197493 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.197467 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d"] Nov 02 18:06:26.198005 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:26.197972 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c963b7_64e0_437c_ade6_6f0c4f7fb3c9.slice/crio-72dcf08f78a944f71d8313c83988e36a821cd494def13a7869b3f0346acd5b62 WatchSource:0}: Error finding container 72dcf08f78a944f71d8313c83988e36a821cd494def13a7869b3f0346acd5b62: Status 404 returned error can't find the container with id 72dcf08f78a944f71d8313c83988e36a821cd494def13a7869b3f0346acd5b62 Nov 02 18:06:26.200686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.200652 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:26.200831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.200719 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:26.200831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.200772 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:06:26.353114 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.353083 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 02 18:06:26.353476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.353150 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 02 18:06:26.699940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.699848 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:26.714761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.710068 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:26.977141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.977046 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" event={"ID":"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9","Type":"ContainerStarted","Data":"8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3"} Nov 02 18:06:26.977141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.977138 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" event={"ID":"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9","Type":"ContainerStarted","Data":"72dcf08f78a944f71d8313c83988e36a821cd494def13a7869b3f0346acd5b62"} Nov 02 18:06:26.977336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.977288 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:26.978321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:26.978302 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:06:27.101431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.101372 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podStartSLOduration=2.101351511 podStartE2EDuration="2.101351511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:27.097573803 +0000 UTC m=+2574.171101008" watchObservedRunningTime="2026-11-02 18:06:27.101351511 +0000 UTC m=+2574.174878715" Nov 02 18:06:27.356072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.356015 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 02 18:06:27.356575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.356194 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 02 18:06:27.742374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.742283 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:27.747668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.747643 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:27.984205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.984169 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:27.985302 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:27.985282 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:06:28.352359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:28.352328 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 02 18:06:28.352530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:28.352379 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 02 18:06:29.353375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:29.353330 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Nov 02 18:06:29.353961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:29.353401 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Nov 02 18:06:30.356476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:30.356439 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:30.357821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:30.357789 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:30.361310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:30.361279 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:06:33.910853 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:33.910814 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:06:33.912264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:33.911926 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:06:35.948150 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:06:35.948107 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:06:39.290761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:39.288908 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 02 18:06:39.290761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:39.289864 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:39.291455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:39.290891 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:39.509761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:39.505954 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Nov 02 18:06:39.596798 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:39.595940 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:06:46.573209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.573174 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d"] Nov 02 18:06:46.578784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.578762 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.635254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.635217 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d"] Nov 02 18:06:46.673845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.673806 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.674028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.673851 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.674028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.673949 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bnd\" (UniqueName: \"kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.674114 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.674031 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.682603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.682577 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-dd92ac-serving-cert\"" Nov 02 18:06:46.775695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.775655 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.775949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.775718 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.775949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.775804 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bnd\" (UniqueName: \"kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.775949 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:06:46.775833 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-serving-cert: secret "maas-api-e2e-shared-a-w1-dd92ac-serving-cert" not found Nov 02 18:06:46.775949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.775863 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.775949 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:06:46.775903 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls podName:fb595908-9b1b-4a55-802e-56b20a8f1ee9 nodeName:}" failed. No retries permitted until 2026-11-02 18:06:47.27588603 +0000 UTC m=+2594.349413210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls") pod "maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" (UID: "fb595908-9b1b-4a55-802e-56b20a8f1ee9") : secret "maas-api-e2e-shared-a-w1-dd92ac-serving-cert" not found Nov 02 18:06:46.776562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.776537 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.778306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.778287 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:46.848160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:46.848094 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bnd\" (UniqueName: \"kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:47.283118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:47.283017 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:47.286004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:47.285967 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:47.493799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:47.493764 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:47.881473 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:06:47.881440 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb595908_9b1b_4a55_802e_56b20a8f1ee9.slice/crio-966171179d039524e9cb8a96d828759fc5b841e99c074d8428cd33f16523926e WatchSource:0}: Error finding container 966171179d039524e9cb8a96d828759fc5b841e99c074d8428cd33f16523926e: Status 404 returned error can't find the container with id 966171179d039524e9cb8a96d828759fc5b841e99c074d8428cd33f16523926e Nov 02 18:06:47.881962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:47.881863 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d"] Nov 02 18:06:48.219637 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:48.219551 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" event={"ID":"fb595908-9b1b-4a55-802e-56b20a8f1ee9","Type":"ContainerStarted","Data":"966171179d039524e9cb8a96d828759fc5b841e99c074d8428cd33f16523926e"} Nov 02 18:06:49.228870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:49.228722 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" event={"ID":"fb595908-9b1b-4a55-802e-56b20a8f1ee9","Type":"ContainerStarted","Data":"09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526"} Nov 02 18:06:49.229275 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:49.228867 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:06:49.348248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:49.348193 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" podStartSLOduration=2.359890082 podStartE2EDuration="3.348178684s" totalImagesPullingTime="988.288602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:06:49.346154327 +0000 UTC m=+2596.419681530" watchObservedRunningTime="2026-11-02 18:06:49.348178684 +0000 UTC m=+2596.421705886" Nov 02 18:06:55.241826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:06:55.241714 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:07:01.511964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:01.511929 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:07:05.956126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:05.956090 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg"] Nov 02 18:07:05.958566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:05.958426 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" podUID="c6d63b5c-a9ab-4ffa-8515-10359bdea516" containerName="payload-pre-processing" containerID="cri-o://67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203" gracePeriod=30 Nov 02 18:07:05.994448 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:05.994410 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:07:06.261361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.261330 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:07:06.293591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.293557 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq\" (UniqueName: \"kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq\") pod \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " Nov 02 18:07:06.293784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.293675 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume\") pod \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\" (UID: \"c6d63b5c-a9ab-4ffa-8515-10359bdea516\") " Nov 02 18:07:06.294307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.294167 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume" pod "c6d63b5c-a9ab-4ffa-8515-10359bdea516" (UID: "c6d63b5c-a9ab-4ffa-8515-10359bdea516"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:06.294307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.294286 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6d63b5c-a9ab-4ffa-8515-10359bdea516-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:06.296641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.296623 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq" pod "c6d63b5c-a9ab-4ffa-8515-10359bdea516" (UID: "c6d63b5c-a9ab-4ffa-8515-10359bdea516"). InnerVolumeSpecName "kube-api-access-pgvdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:06.367155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.366979 5647 generic.go:356] "Generic (PLEG): container finished" podID="c6d63b5c-a9ab-4ffa-8515-10359bdea516" containerID="67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203" exitCode=0 Nov 02 18:07:06.367155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.367034 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" Nov 02 18:07:06.367155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.367060 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" event={"ID":"c6d63b5c-a9ab-4ffa-8515-10359bdea516","Type":"ContainerDied","Data":"67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203"} Nov 02 18:07:06.367155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.367113 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg" event={"ID":"c6d63b5c-a9ab-4ffa-8515-10359bdea516","Type":"ContainerDied","Data":"bc61587127d8cf86748c3f060a3146d46306eaf181d190b0ce5ad3eab85d7b7e"} Nov 02 18:07:06.367155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.367138 5647 scope.go:122] "RemoveContainer" containerID="67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203" Nov 02 18:07:06.429783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.427418 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgvdq\" (UniqueName: \"kubernetes.io/projected/c6d63b5c-a9ab-4ffa-8515-10359bdea516-kube-api-access-pgvdq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:06.501289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.501166 5647 scope.go:122] "RemoveContainer" containerID="67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203" Nov 02 18:07:06.501918 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:06.501838 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203\": container with ID starting with 67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203 not found: ID does not exist" containerID="67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203" Nov 02 18:07:06.501918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.501881 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203"} err="failed to get container status \"67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203\": rpc error: code = NotFound desc = could not find container \"67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203\": container with ID starting with 67348e385f0fd83ddb0c6b7cfeb29b9b9b39f720f92f49e4402457c8d4881203 not found: ID does not exist" Nov 02 18:07:06.591325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.591281 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg"] Nov 02 18:07:06.595574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:06.595542 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f07b5b-855df669b7-tqdbg"] Nov 02 18:07:07.081704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.081674 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d63b5c-a9ab-4ffa-8515-10359bdea516" path="/var/lib/kubelet/pods/c6d63b5c-a9ab-4ffa-8515-10359bdea516/volumes" Nov 02 18:07:07.082143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.081996 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss"] Nov 02 18:07:07.082512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.082497 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c6d63b5c-a9ab-4ffa-8515-10359bdea516" containerName="payload-pre-processing" Nov 02 18:07:07.082552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.082514 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d63b5c-a9ab-4ffa-8515-10359bdea516" containerName="payload-pre-processing" Nov 02 18:07:07.082646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.082636 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d63b5c-a9ab-4ffa-8515-10359bdea516" containerName="payload-pre-processing" Nov 02 18:07:07.087919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.087897 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.150810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.141804 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-580a21-serving-cert\"" Nov 02 18:07:07.162812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.159757 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss"] Nov 02 18:07:07.244358 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.244312 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2rlk\" (UniqueName: \"kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.244582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.244428 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.244582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.244523 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.244766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.244744 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.349141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.348712 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rlk\" (UniqueName: \"kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.363195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.363155 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.363595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.363575 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.363921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.363889 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.364581 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:07.363446 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-serving-cert: secret "maas-api-e2e-shared-a-w0-580a21-serving-cert" not found Nov 02 18:07:07.364808 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:07.364792 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls podName:25931f24-3219-4adb-a7bf-062e058b5ea1 nodeName:}" failed. No retries permitted until 2026-11-02 18:07:07.864761652 +0000 UTC m=+2614.938288852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls") pod "maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" (UID: "25931f24-3219-4adb-a7bf-062e058b5ea1") : secret "maas-api-e2e-shared-a-w0-580a21-serving-cert" not found Nov 02 18:07:07.371455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.365870 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.386485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.386440 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.434803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.423204 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rlk\" (UniqueName: \"kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.870601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.870564 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:07.873415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:07.873383 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:08.002469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:08.002376 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:08.391444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:08.391407 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss"] Nov 02 18:07:09.403424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:09.403384 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" event={"ID":"25931f24-3219-4adb-a7bf-062e058b5ea1","Type":"ContainerStarted","Data":"48869a7356ce8a2d755f6542837669162ec3f9ed51cd4495b1f1231f8b32d953"} Nov 02 18:07:09.403905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:09.403446 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" event={"ID":"25931f24-3219-4adb-a7bf-062e058b5ea1","Type":"ContainerStarted","Data":"6ce60460bb469ad7c10d1613de3c2c472c9921695d853abcef9b175e8fa69efd"} Nov 02 18:07:09.403905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:09.403599 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:09.533659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:09.533559 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4"] Nov 02 18:07:09.533949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:09.533813 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" containerID="cri-o://d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d" gracePeriod=30 Nov 02 18:07:09.565905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:09.565850 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" podStartSLOduration=1.69878589 podStartE2EDuration="2.565835671s" totalImagesPullingTime="867.049781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:09.559946601 +0000 UTC m=+2616.633473803" watchObservedRunningTime="2026-11-02 18:07:09.565835671 +0000 UTC m=+2616.639362873" Nov 02 18:07:11.430960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.430841 5647 generic.go:356] "Generic (PLEG): container finished" podID="59bd3539-3194-4879-87b9-48a316cda7eb" containerID="e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14" exitCode=0 Nov 02 18:07:11.430960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.430917 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" event={"ID":"59bd3539-3194-4879-87b9-48a316cda7eb","Type":"ContainerDied","Data":"e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14"} Nov 02 18:07:11.431455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.431352 5647 scope.go:122] "RemoveContainer" containerID="e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14" Nov 02 18:07:11.432978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.432932 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:11.433113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.433029 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:11.433113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.433077 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:11.448276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.448248 5647 patch_prober.go:29] interesting pod/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:07:11.448347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:11.448323 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:07:12.443959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:12.443921 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" event={"ID":"59bd3539-3194-4879-87b9-48a316cda7eb","Type":"ContainerStarted","Data":"a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee"} Nov 02 18:07:12.491366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:12.491335 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:07:12.491565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:12.491500 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 02 18:07:12.491565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:12.491549 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 02 18:07:13.491498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:13.491458 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 02 18:07:13.491970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:13.491520 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 02 18:07:14.491554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.491519 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 02 18:07:14.492125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.491580 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 02 18:07:14.859479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.859451 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 18:07:14.928341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928297 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928346 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928392 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928445 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928497 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928530 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928595 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928666 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:14.928926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928699 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928757 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo\") pod \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\" (UID: \"52cd3067-1b2c-492c-a7d8-e28d39754a5a\") " Nov 02 18:07:14.928926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928814 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:14.928926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.928844 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:14.929207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929000 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:14.929207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929130 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:14.929447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929372 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:14.929447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929394 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:14.929447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929408 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:14.929447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929419 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:14.929447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.929428 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:14.931269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.931240 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:14.931399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.931380 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:14.931561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.931542 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:14.931627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:14.931570 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw" pod "52cd3067-1b2c-492c-a7d8-e28d39754a5a" (UID: "52cd3067-1b2c-492c-a7d8-e28d39754a5a"). InnerVolumeSpecName "kube-api-access-rjvbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:15.031003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.030966 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjvbw\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-kube-api-access-rjvbw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:15.031003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.030996 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:15.031003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.031006 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:15.031268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.031016 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52cd3067-1b2c-492c-a7d8-e28d39754a5a-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:15.431068 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.430995 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:07:15.477336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.477300 5647 generic.go:356] "Generic (PLEG): container finished" podID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerID="d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d" exitCode=0 Nov 02 18:07:15.477530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.477353 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" Nov 02 18:07:15.477530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.477369 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" event={"ID":"52cd3067-1b2c-492c-a7d8-e28d39754a5a","Type":"ContainerDied","Data":"d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d"} Nov 02 18:07:15.477530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.477409 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4" event={"ID":"52cd3067-1b2c-492c-a7d8-e28d39754a5a","Type":"ContainerDied","Data":"56e71ce2e47717ebeabdb727cfe99e213f5ba39cb7df827095e1dd2e6ab4b7ba"} Nov 02 18:07:15.477530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.477431 5647 scope.go:122] "RemoveContainer" containerID="d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d" Nov 02 18:07:15.487617 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.487597 5647 scope.go:122] "RemoveContainer" containerID="f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9" Nov 02 18:07:15.491747 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.491701 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 02 18:07:15.492109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.491782 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 02 18:07:15.497842 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.497822 5647 scope.go:122] "RemoveContainer" containerID="d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d" Nov 02 18:07:15.498139 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:15.498113 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d\": container with ID starting with d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d not found: ID does not exist" containerID="d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d" Nov 02 18:07:15.498201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.498153 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d"} err="failed to get container status \"d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d\": rpc error: code = NotFound desc = could not find container \"d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d\": container with ID starting with d6a8c8105fcd4c4f72750db3c8bbdd1b4de39314aa15e246f546a4e13dfcfe6d not found: ID does not exist" Nov 02 18:07:15.498201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.498178 5647 scope.go:122] "RemoveContainer" containerID="f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9" Nov 02 18:07:15.498492 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:15.498471 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9\": container with ID starting with f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9 not found: ID does not exist" containerID="f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9" Nov 02 18:07:15.498562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.498501 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9"} err="failed to get container status \"f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9\": rpc error: code = NotFound desc = could not find container \"f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9\": container with ID starting with f9a1979e1ec8c9c421e07a72cc9710bb9a69dd3cf2d259b2cb72d2f641b483a9 not found: ID does not exist" Nov 02 18:07:15.675285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.675257 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4"] Nov 02 18:07:15.678316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:15.678285 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f07b5b-openshift-default-5cb8db9d79-79sm4"] Nov 02 18:07:16.492148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:16.492115 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 02 18:07:16.492554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:16.492169 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 02 18:07:17.081877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.081841 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" path="/var/lib/kubelet/pods/52cd3067-1b2c-492c-a7d8-e28d39754a5a/volumes" Nov 02 18:07:17.492065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.492032 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Nov 02 18:07:17.492263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.492092 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Nov 02 18:07:17.686441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.686400 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm"] Nov 02 18:07:17.687058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.687038 5647 cpu_manager.go:414] "containerMap: removing container" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" Nov 02 18:07:17.687058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.687057 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" Nov 02 18:07:17.687234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.687076 5647 cpu_manager.go:414] "containerMap: removing container" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" Nov 02 18:07:17.687234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.687081 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" Nov 02 18:07:17.687234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.687164 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" Nov 02 18:07:17.690445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.690416 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.743627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.743487 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f7aa2e-serving-cert\"" Nov 02 18:07:17.764525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.764491 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.764715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.764682 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.764887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.764864 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.764943 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.764910 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssmj\" (UniqueName: \"kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.798075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.798031 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm"] Nov 02 18:07:17.865905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.865867 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssmj\" (UniqueName: \"kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.866084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.866023 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.866137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.866119 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.866237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.866222 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.866287 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:17.866265 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-serving-cert: secret "maas-api-e2e-shared-a-w3-f7aa2e-serving-cert" not found Nov 02 18:07:17.866343 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:17.866333 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls podName:93d41eb7-accd-4b43-a7c7-121e38a885d8 nodeName:}" failed. No retries permitted until 2026-11-02 18:07:18.366316108 +0000 UTC m=+2625.439843289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls") pod "maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" (UID: "93d41eb7-accd-4b43-a7c7-121e38a885d8") : secret "maas-api-e2e-shared-a-w3-f7aa2e-serving-cert" not found Nov 02 18:07:17.866635 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.866615 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.868798 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.868781 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:17.964760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:17.962926 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssmj\" (UniqueName: \"kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:18.375596 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.375559 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:18.378251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.378220 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:18.497685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.497649 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:07:18.498258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.498230 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:07:18.500624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.500603 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:07:18.604261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.604229 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:18.989265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:18.989233 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm"] Nov 02 18:07:18.989750 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:07:18.989711 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d41eb7_accd_4b43_a7c7_121e38a885d8.slice/crio-c37b87f06efac1e1c4e840256f915703041fce6ae9d51baa72e440cc208efa6f WatchSource:0}: Error finding container c37b87f06efac1e1c4e840256f915703041fce6ae9d51baa72e440cc208efa6f: Status 404 returned error can't find the container with id c37b87f06efac1e1c4e840256f915703041fce6ae9d51baa72e440cc208efa6f Nov 02 18:07:19.508223 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:19.508184 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" event={"ID":"93d41eb7-accd-4b43-a7c7-121e38a885d8","Type":"ContainerStarted","Data":"c37b87f06efac1e1c4e840256f915703041fce6ae9d51baa72e440cc208efa6f"} Nov 02 18:07:20.517041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:20.517009 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" event={"ID":"93d41eb7-accd-4b43-a7c7-121e38a885d8","Type":"ContainerStarted","Data":"0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950"} Nov 02 18:07:20.517477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:20.517076 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:20.636245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:20.636185 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" podStartSLOduration=2.736301403 podStartE2EDuration="3.636169489s" totalImagesPullingTime="899.868086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:20.634751774 +0000 UTC m=+2627.708278968" watchObservedRunningTime="2026-11-02 18:07:20.636169489 +0000 UTC m=+2627.709696691" Nov 02 18:07:26.530125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:26.530091 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:07:31.367341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:31.367301 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:07:35.973514 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:07:35.973467 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:07:37.270839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.270805 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw"] Nov 02 18:07:37.271313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.271066 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" podUID="21237a61-6e71-48d7-b08e-dc6e4b3ebeab" containerName="maas-api" containerID="cri-o://a5359a68bf78250b46afbd79b6bde680e89adff4c1fe78511a0a5229e2a8ce1d" gracePeriod=30 Nov 02 18:07:37.734675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.734622 5647 generic.go:356] "Generic (PLEG): container finished" podID="21237a61-6e71-48d7-b08e-dc6e4b3ebeab" containerID="a5359a68bf78250b46afbd79b6bde680e89adff4c1fe78511a0a5229e2a8ce1d" exitCode=0 Nov 02 18:07:37.735090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.735050 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" event={"ID":"21237a61-6e71-48d7-b08e-dc6e4b3ebeab","Type":"ContainerDied","Data":"a5359a68bf78250b46afbd79b6bde680e89adff4c1fe78511a0a5229e2a8ce1d"} Nov 02 18:07:37.778584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.778554 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:07:37.891508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.891472 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8\" (UniqueName: \"kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8\") pod \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " Nov 02 18:07:37.891508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.891515 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca\") pod \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " Nov 02 18:07:37.891875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.891848 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs\") pod \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " Nov 02 18:07:37.892129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.891917 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\") pod \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\" (UID: \"21237a61-6e71-48d7-b08e-dc6e4b3ebeab\") " Nov 02 18:07:37.892129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.891951 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca" pod "21237a61-6e71-48d7-b08e-dc6e4b3ebeab" (UID: "21237a61-6e71-48d7-b08e-dc6e4b3ebeab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:37.892618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.892592 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:37.894394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.894348 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8" pod "21237a61-6e71-48d7-b08e-dc6e4b3ebeab" (UID: "21237a61-6e71-48d7-b08e-dc6e4b3ebeab"). InnerVolumeSpecName "kube-api-access-vvmj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:37.894669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.894621 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs" pod "21237a61-6e71-48d7-b08e-dc6e4b3ebeab" (UID: "21237a61-6e71-48d7-b08e-dc6e4b3ebeab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:07:37.894805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.894670 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls" pod "21237a61-6e71-48d7-b08e-dc6e4b3ebeab" (UID: "21237a61-6e71-48d7-b08e-dc6e4b3ebeab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:07:37.994035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.993994 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:37.994035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.994035 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:37.994248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:37.994054 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmj8\" (UniqueName: \"kubernetes.io/projected/21237a61-6e71-48d7-b08e-dc6e4b3ebeab-kube-api-access-vvmj8\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:38.744681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:38.744646 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" event={"ID":"21237a61-6e71-48d7-b08e-dc6e4b3ebeab","Type":"ContainerDied","Data":"604f953a06a9059e838323b0ad7008fac332b24e2516aa544e3f7f51bdedb24e"} Nov 02 18:07:38.745221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:38.744659 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw" Nov 02 18:07:38.745221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:38.744716 5647 scope.go:122] "RemoveContainer" containerID="a5359a68bf78250b46afbd79b6bde680e89adff4c1fe78511a0a5229e2a8ce1d" Nov 02 18:07:38.919466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:38.919421 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw"] Nov 02 18:07:38.970565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:38.970531 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-948451-df7574b69-wknqw"] Nov 02 18:07:39.085450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:39.085365 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21237a61-6e71-48d7-b08e-dc6e4b3ebeab" path="/var/lib/kubelet/pods/21237a61-6e71-48d7-b08e-dc6e4b3ebeab/volumes" Nov 02 18:07:41.978828 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:41.978793 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 02 18:07:41.979263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:41.978854 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 02 18:07:42.694435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.694399 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h"] Nov 02 18:07:42.694680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.694644 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" containerID="cri-o://7dba64d62b455cc06410a37f540fc4bfcf1050d56575a2a2cc2f489c289c81dc" gracePeriod=30 Nov 02 18:07:42.860111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.860067 5647 generic.go:356] "Generic (PLEG): container finished" podID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerID="466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5" exitCode=0 Nov 02 18:07:42.860672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.860454 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" event={"ID":"54da1589-034c-48a5-b9dc-b10a59c90b76","Type":"ContainerDied","Data":"466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5"} Nov 02 18:07:42.867597 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.867564 5647 generic.go:356] "Generic (PLEG): container finished" podID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerID="92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1" exitCode=0 Nov 02 18:07:42.868086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.867922 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" event={"ID":"6e44a8e2-8ff6-466d-a172-dd06de373029","Type":"ContainerDied","Data":"92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1"} Nov 02 18:07:42.872553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.872528 5647 scope.go:122] "RemoveContainer" containerID="92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1" Nov 02 18:07:42.880708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.880646 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:42.881021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.880978 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:42.881606 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.881578 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:42.885486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.885463 5647 scope.go:122] "RemoveContainer" containerID="466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5" Nov 02 18:07:42.913825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.913756 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:42.914139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.914105 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:42.914283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:42.914174 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:43.885395 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:43.885357 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" event={"ID":"54da1589-034c-48a5-b9dc-b10a59c90b76","Type":"ContainerStarted","Data":"c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4"} Nov 02 18:07:43.888977 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:43.888945 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" event={"ID":"6e44a8e2-8ff6-466d-a172-dd06de373029","Type":"ContainerStarted","Data":"dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc"} Nov 02 18:07:44.039397 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.039357 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:07:44.043766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.039876 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 02 18:07:44.043766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.039958 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 02 18:07:44.699675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.699639 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:07:44.699956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.699907 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 02 18:07:44.700037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.699961 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 02 18:07:44.761906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.761871 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44"] Nov 02 18:07:44.765786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.763256 5647 cpu_manager.go:414] "containerMap: removing container" podUID="21237a61-6e71-48d7-b08e-dc6e4b3ebeab" containerName="maas-api" Nov 02 18:07:44.765786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.763293 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21237a61-6e71-48d7-b08e-dc6e4b3ebeab" containerName="maas-api" Nov 02 18:07:44.765786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.763651 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cd3067-1b2c-492c-a7d8-e28d39754a5a" containerName="istio-proxy" Nov 02 18:07:44.765786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.763673 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="21237a61-6e71-48d7-b08e-dc6e4b3ebeab" containerName="maas-api" Nov 02 18:07:44.770583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.767088 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.827761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.827433 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-eb1633-openshift-default-dockercfg-fx2vp\"" Nov 02 18:07:44.872531 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.872485 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44"] Nov 02 18:07:44.877046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877010 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877130 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877152 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877196 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877284 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877334 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877414 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnj6\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877544 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:44.877624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:44.877573 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.136928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.131140 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 02 18:07:45.136928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.131214 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.114776 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.192896 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.192930 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.193020 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.193049 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.193082 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.193246 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.193339 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.193390 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnj6\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.196793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.196188 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.197559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.196883 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.201033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.197763 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.225760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.215151 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.225760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.215949 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.255757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.255696 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.268792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.256413 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.268792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.260088 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.278669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.278635 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnj6\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6\") pod \"e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.385024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.384988 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:45.656942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.656819 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:07:45.701400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.701237 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 02 18:07:45.701400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.701284 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 02 18:07:45.871281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.871253 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44"] Nov 02 18:07:45.872800 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:07:45.872771 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9768201e_ace8_4671_8526_64bd24fe8aa5.slice/crio-c535d39756a6212c4d758d483a429e8a7a6b6a2426a8aef3507254f288160bbf WatchSource:0}: Error finding container c535d39756a6212c4d758d483a429e8a7a6b6a2426a8aef3507254f288160bbf: Status 404 returned error can't find the container with id c535d39756a6212c4d758d483a429e8a7a6b6a2426a8aef3507254f288160bbf Nov 02 18:07:45.875499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.875442 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:45.875599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.875540 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:45.875599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.875578 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:07:45.958013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:45.957973 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" event={"ID":"9768201e-ace8-4671-8526-64bd24fe8aa5","Type":"ContainerStarted","Data":"c535d39756a6212c4d758d483a429e8a7a6b6a2426a8aef3507254f288160bbf"} Nov 02 18:07:46.037803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:46.037245 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 02 18:07:46.037803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:46.037315 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 02 18:07:46.700836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:46.700800 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 02 18:07:46.701252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:46.700850 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 02 18:07:47.065815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.065389 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" event={"ID":"9768201e-ace8-4671-8526-64bd24fe8aa5","Type":"ContainerStarted","Data":"0c8f31081d6f7f537a4bfa4af71b11c68a84fd89744605289ce12adb643657e5"} Nov 02 18:07:47.101764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.066634 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 02 18:07:47.101764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.066685 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 02 18:07:47.343954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.343875 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podStartSLOduration=3.343852679 podStartE2EDuration="3.343852679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:47.339105674 +0000 UTC m=+2654.412632902" watchObservedRunningTime="2026-11-02 18:07:47.343852679 +0000 UTC m=+2654.417379884" Nov 02 18:07:47.385848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.385816 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:47.391089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.391063 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:47.700881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.700503 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 02 18:07:47.700881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:47.700565 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 02 18:07:48.053768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.042214 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 02 18:07:48.053768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.042299 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 02 18:07:48.213756 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.213641 5647 generic.go:356] "Generic (PLEG): container finished" podID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerID="7dba64d62b455cc06410a37f540fc4bfcf1050d56575a2a2cc2f489c289c81dc" exitCode=0 Nov 02 18:07:48.216115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.216080 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" event={"ID":"080fb4bf-8385-4e94-a757-adf3b771f4fa","Type":"ContainerDied","Data":"7dba64d62b455cc06410a37f540fc4bfcf1050d56575a2a2cc2f489c289c81dc"} Nov 02 18:07:48.216390 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.216371 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:48.216552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.216537 5647 scope.go:122] "RemoveContainer" containerID="979deb33e252e5b39dd1304aab8416b050249f73521bb8c9590af67686213b43" Nov 02 18:07:48.218909 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.218886 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:07:48.264836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.264802 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:07:48.396752 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.396695 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.396964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.396788 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.396964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.396832 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.396964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.396872 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.396961 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397002 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397022 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397031 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397072 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397111 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.397149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397143 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:07:48.397514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397166 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:48.397514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397218 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:48.397514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397235 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo\") pod \"080fb4bf-8385-4e94-a757-adf3b771f4fa\" (UID: \"080fb4bf-8385-4e94-a757-adf3b771f4fa\") " Nov 02 18:07:48.397781 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397757 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.397919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397784 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.397919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397800 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080fb4bf-8385-4e94-a757-adf3b771f4fa-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.397919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397815 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.397919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.397830 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.399289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.399267 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "kube-api-access-xq7sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:48.399481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.399461 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:07:48.399766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.399708 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:07:48.399876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.399779 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token" pod "080fb4bf-8385-4e94-a757-adf3b771f4fa" (UID: "080fb4bf-8385-4e94-a757-adf3b771f4fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:07:48.499078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.498977 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq7sd\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-kube-api-access-xq7sd\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.499078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.499019 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.499078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.499034 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.499078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.499047 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080fb4bf-8385-4e94-a757-adf3b771f4fa-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:07:48.700904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.700865 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Nov 02 18:07:48.701333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:48.700916 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Nov 02 18:07:49.053189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.053148 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 02 18:07:49.054441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.054408 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 02 18:07:49.261716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.261673 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" event={"ID":"080fb4bf-8385-4e94-a757-adf3b771f4fa","Type":"ContainerDied","Data":"7e2ed35737273fc444a4657412b86ce4fad8c9ce14a6d418d6bc9f2d37a4b9f0"} Nov 02 18:07:49.261964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.261839 5647 scope.go:122] "RemoveContainer" containerID="7dba64d62b455cc06410a37f540fc4bfcf1050d56575a2a2cc2f489c289c81dc" Nov 02 18:07:49.261964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.261693 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h" Nov 02 18:07:49.506785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.506724 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h"] Nov 02 18:07:49.509885 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.509860 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-948451-openshift-default-6769dd947f-jmm7h"] Nov 02 18:07:49.794040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.793935 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:07:49.794491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.794351 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:07:49.795339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:49.795315 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:07:50.066845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:50.065569 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:07:50.068600 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:50.068079 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:07:51.082254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:51.082219 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" path="/var/lib/kubelet/pods/080fb4bf-8385-4e94-a757-adf3b771f4fa/volumes" Nov 02 18:07:54.102857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.102811 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p"] Nov 02 18:07:54.103805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.103782 5647 cpu_manager.go:414] "containerMap: removing container" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" Nov 02 18:07:54.103870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.103808 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" Nov 02 18:07:54.103870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.103832 5647 cpu_manager.go:414] "containerMap: removing container" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" Nov 02 18:07:54.103870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.103841 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" Nov 02 18:07:54.104048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.104033 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" Nov 02 18:07:54.104082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.104055 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="080fb4bf-8385-4e94-a757-adf3b771f4fa" containerName="istio-proxy" Nov 02 18:07:54.109407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.109385 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.162597 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.162563 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p"] Nov 02 18:07:54.209309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.209283 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-eb1633\"" Nov 02 18:07:54.209309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.209285 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-eb1633-dockercfg-vpd64\"" Nov 02 18:07:54.220758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.220704 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.220951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.220935 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9pgc\" (UniqueName: \"kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc\") pod \"payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.322056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.322010 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9pgc\" (UniqueName: \"kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc\") pod \"payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.322273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.322070 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.322833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.322811 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.381266 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.381182 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9pgc\" (UniqueName: \"kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc\") pod \"payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.423095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.423053 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:54.802405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:54.802366 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p"] Nov 02 18:07:54.802934 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:07:54.802906 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a757d14_821c_48e7_8c05_beb593e97dc0.slice/crio-a86cb42f073f3d1b836cddacbea1f1fbca593103362aecb62736129386a798c2 WatchSource:0}: Error finding container a86cb42f073f3d1b836cddacbea1f1fbca593103362aecb62736129386a798c2: Status 404 returned error can't find the container with id a86cb42f073f3d1b836cddacbea1f1fbca593103362aecb62736129386a798c2 Nov 02 18:07:55.310322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:55.310284 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" event={"ID":"3a757d14-821c-48e7-8c05-beb593e97dc0","Type":"ContainerStarted","Data":"d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55"} Nov 02 18:07:55.310782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:55.310350 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:07:55.310782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:55.310372 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" event={"ID":"3a757d14-821c-48e7-8c05-beb593e97dc0","Type":"ContainerStarted","Data":"a86cb42f073f3d1b836cddacbea1f1fbca593103362aecb62736129386a798c2"} Nov 02 18:07:55.430297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:55.430236 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" podStartSLOduration=1.430222845 podStartE2EDuration="1.430222845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:07:55.427570915 +0000 UTC m=+2662.501098117" watchObservedRunningTime="2026-11-02 18:07:55.430222845 +0000 UTC m=+2662.503750048" Nov 02 18:07:58.749028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.748987 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2"] Nov 02 18:07:58.753194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.753172 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:58.783201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.783156 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:58.783379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.783222 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqz64\" (UniqueName: \"kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64\") pod \"payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:58.813496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.813454 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2"] Nov 02 18:07:58.868787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.861162 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-4ecad8-dockercfg-kx6cg\"" Nov 02 18:07:58.876759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.869203 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-4ecad8\"" Nov 02 18:07:58.933778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.933725 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:58.951118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.934953 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:58.951869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.951383 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqz64\" (UniqueName: \"kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64\") pod \"payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:58.974612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.974573 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5"] Nov 02 18:07:58.981613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.981582 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:58.992209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:58.991675 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5"] Nov 02 18:07:59.010768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.010717 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqz64\" (UniqueName: \"kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64\") pod \"payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:59.067300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.067269 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:07:59.161097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.161065 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.161298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.161181 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6vdc\" (UniqueName: \"kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc\") pod \"payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.262313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.262204 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6vdc\" (UniqueName: \"kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc\") pod \"payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.262516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.262363 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.263189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.263158 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.321567 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.321535 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6vdc\" (UniqueName: \"kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc\") pod \"payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.446752 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.446706 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2"] Nov 02 18:07:59.447368 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:07:59.447343 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd21626f_0d9e_43db_9cbc_953645494d5f.slice/crio-4e38d64653e04cef5a514c26ba26664b270879c559cc4493a95d2544fda0b9e8 WatchSource:0}: Error finding container 4e38d64653e04cef5a514c26ba26664b270879c559cc4493a95d2544fda0b9e8: Status 404 returned error can't find the container with id 4e38d64653e04cef5a514c26ba26664b270879c559cc4493a95d2544fda0b9e8 Nov 02 18:07:59.618278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.612994 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:07:59.870275 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:07:59.870241 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:08:00.110430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.110398 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5"] Nov 02 18:08:00.111462 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:08:00.111440 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe81d0ac_2dd1_4e66_bfc0_c89ecc5ed1cc.slice/crio-86878bfd0a278fcd90106439d91ccfc1b854257c0c353b0fd2f1d9b93d8e6cfb WatchSource:0}: Error finding container 86878bfd0a278fcd90106439d91ccfc1b854257c0c353b0fd2f1d9b93d8e6cfb: Status 404 returned error can't find the container with id 86878bfd0a278fcd90106439d91ccfc1b854257c0c353b0fd2f1d9b93d8e6cfb Nov 02 18:08:00.345037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.344992 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" event={"ID":"dd21626f-0d9e-43db-9cbc-953645494d5f","Type":"ContainerStarted","Data":"cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b"} Nov 02 18:08:00.345270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.345105 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:08:00.345270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.345130 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" event={"ID":"dd21626f-0d9e-43db-9cbc-953645494d5f","Type":"ContainerStarted","Data":"4e38d64653e04cef5a514c26ba26664b270879c559cc4493a95d2544fda0b9e8"} Nov 02 18:08:00.346986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.346960 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" event={"ID":"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc","Type":"ContainerStarted","Data":"70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4"} Nov 02 18:08:00.347106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.346999 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" event={"ID":"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc","Type":"ContainerStarted","Data":"86878bfd0a278fcd90106439d91ccfc1b854257c0c353b0fd2f1d9b93d8e6cfb"} Nov 02 18:08:00.347106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.347033 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:08:00.465269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.465155 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" podStartSLOduration=2.465142752 podStartE2EDuration="2.465142752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:00.461941628 +0000 UTC m=+2667.535468833" watchObservedRunningTime="2026-11-02 18:08:00.465142752 +0000 UTC m=+2667.538669985" Nov 02 18:08:00.588759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:00.587790 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" podStartSLOduration=2.587769041 podStartE2EDuration="2.587769041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:00.582118033 +0000 UTC m=+2667.655645237" watchObservedRunningTime="2026-11-02 18:08:00.587769041 +0000 UTC m=+2667.661296245" Nov 02 18:08:05.953395 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:08:05.953346 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:08:06.327782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:06.327664 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:08:10.421048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.421012 5647 generic.go:356] "Generic (PLEG): container finished" podID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerID="8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3" exitCode=0 Nov 02 18:08:10.421518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.421065 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" event={"ID":"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9","Type":"ContainerDied","Data":"8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3"} Nov 02 18:08:10.421518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.421489 5647 scope.go:122] "RemoveContainer" containerID="8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3" Nov 02 18:08:10.423313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.423273 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:10.423400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.423374 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:10.423442 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.423426 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:10.657290 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.657208 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4"] Nov 02 18:08:10.662636 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.662609 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.736206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.736175 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-0ad26a-openshift-default-dockercfg-hbjsj\"" Nov 02 18:08:10.740906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.740859 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4"] Nov 02 18:08:10.759675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.759643 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mw4r\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.759912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.759689 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.759912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.759831 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.759912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.759884 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.760118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.760009 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.760398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.760195 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.760398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.760274 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.761105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.760442 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.761105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.760477 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862200 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862243 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862289 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862379 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862401 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862475 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862492 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862511 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mw4r\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862554 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862763 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.862991 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.862954 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.863065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.863040 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.863188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.863165 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.863404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.863355 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.865219 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.865188 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.865219 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.865192 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.941820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.941452 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.945756 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.945528 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mw4r\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r\") pod \"e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:10.981778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:10.980977 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:11.356535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.356502 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:08:11.356764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.356575 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:08:11.433235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.433199 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" event={"ID":"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9","Type":"ContainerStarted","Data":"6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2"} Nov 02 18:08:11.540907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.540876 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4"] Nov 02 18:08:11.541386 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:08:11.541358 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007f9dc9_1629_4cc5_9f79_5355b5c7f11f.slice/crio-d1ac60aa524ecdd53fdccec787e0ebbbd543844ebde00d7c4468b0527765f857 WatchSource:0}: Error finding container d1ac60aa524ecdd53fdccec787e0ebbbd543844ebde00d7c4468b0527765f857: Status 404 returned error can't find the container with id d1ac60aa524ecdd53fdccec787e0ebbbd543844ebde00d7c4468b0527765f857 Nov 02 18:08:11.544151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.544116 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:11.544248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.544181 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:11.544248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.544210 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:11.746288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.746121 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:08:11.746955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.746932 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 02 18:08:11.747083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:11.747006 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 02 18:08:12.441920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.441875 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" event={"ID":"007f9dc9-1629-4cc5-9f79-5355b5c7f11f","Type":"ContainerStarted","Data":"c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a"} Nov 02 18:08:12.442407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.441966 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" event={"ID":"007f9dc9-1629-4cc5-9f79-5355b5c7f11f","Type":"ContainerStarted","Data":"d1ac60aa524ecdd53fdccec787e0ebbbd543844ebde00d7c4468b0527765f857"} Nov 02 18:08:12.566501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.566444 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podStartSLOduration=2.566428808 podStartE2EDuration="2.566428808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:12.563520687 +0000 UTC m=+2679.637047890" watchObservedRunningTime="2026-11-02 18:08:12.566428808 +0000 UTC m=+2679.639956008" Nov 02 18:08:12.747820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.743142 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 02 18:08:12.747820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.743208 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 02 18:08:12.981776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.981741 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:12.984165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.984137 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:08:12.987142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:12.987119 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:13.449134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:13.449083 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:13.450156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:13.450131 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:08:13.742365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:13.742274 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 02 18:08:13.742365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:13.742346 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 02 18:08:14.742079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:14.742044 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 02 18:08:14.742513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:14.742103 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 02 18:08:15.745092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:15.745057 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 02 18:08:15.745813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:15.745778 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 02 18:08:16.742627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:16.742575 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Nov 02 18:08:16.742870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:16.742673 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Nov 02 18:08:17.746820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:17.746789 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:08:17.747951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:17.747931 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:08:21.051945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:21.051905 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j"] Nov 02 18:08:21.102726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:21.102689 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7811dfc0-9j77j"] Nov 02 18:08:23.081920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:23.081886 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc759a51-cbd6-4917-b050-806dc5481899" path="/var/lib/kubelet/pods/cc759a51-cbd6-4917-b050-806dc5481899/volumes" Nov 02 18:08:27.992067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:27.992030 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9"] Nov 02 18:08:27.997859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:27.997834 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.060195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.060158 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c82472-openshift-default-dockercfg-kr6zx\"" Nov 02 18:08:28.061746 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.061688 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.062031 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.061996 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.062322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.062292 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.062571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.062550 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.062713 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.062694 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.062859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.062843 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.062988 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.062974 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.079761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.063088 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7nph\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.080123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.080092 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.196978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.188567 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.198589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.198557 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.198830 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.198813 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.198966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.198950 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.199100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.199086 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.199247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.199230 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7nph\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.199414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.199393 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.199722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.199706 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.199881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.199867 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.200507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.200477 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.201046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.201020 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.202721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.202552 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.202721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.202674 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.215836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.209336 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9"] Nov 02 18:08:28.215836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.210101 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.215836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.210958 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.218870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.216754 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.292058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.291970 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7nph\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.292871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.292836 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token\") pod \"e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.315339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.315301 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:28.801458 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:08:28.801425 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0324d641_3db4_4347_93ff_f68b9fb6f913.slice/crio-c6093238705e069322c84a563b96d4fb23349dfc994ac236b6ac3ee80ec7819b WatchSource:0}: Error finding container c6093238705e069322c84a563b96d4fb23349dfc994ac236b6ac3ee80ec7819b: Status 404 returned error can't find the container with id c6093238705e069322c84a563b96d4fb23349dfc994ac236b6ac3ee80ec7819b Nov 02 18:08:28.801646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.801624 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9"] Nov 02 18:08:28.804211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.804176 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:28.804299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.804246 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:28.804299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:28.804275 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:08:29.611591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:29.611557 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" event={"ID":"0324d641-3db4-4347-93ff-f68b9fb6f913","Type":"ContainerStarted","Data":"4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a"} Nov 02 18:08:29.612118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:29.611628 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" event={"ID":"0324d641-3db4-4347-93ff-f68b9fb6f913","Type":"ContainerStarted","Data":"c6093238705e069322c84a563b96d4fb23349dfc994ac236b6ac3ee80ec7819b"} Nov 02 18:08:29.786621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:29.786564 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podStartSLOduration=2.786548997 podStartE2EDuration="2.786548997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:08:29.782428017 +0000 UTC m=+2696.855955221" watchObservedRunningTime="2026-11-02 18:08:29.786548997 +0000 UTC m=+2696.860076277" Nov 02 18:08:30.316149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:30.316112 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:30.322151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:30.322123 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:30.619122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:30.619094 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:30.620155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:30.620134 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:08:34.318162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:34.318125 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:08:34.320398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:08:34.320375 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:08:35.998762 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:08:35.988183 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:09:05.966807 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:09:05.966765 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:09:05.968221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:05.968197 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:09:15.047726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.047675 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p"] Nov 02 18:09:15.098084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.098032 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96c1ea65-x8h9p"] Nov 02 18:09:15.875545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.875504 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5"] Nov 02 18:09:15.880091 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.880067 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:15.937017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.936986 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5"] Nov 02 18:09:15.937844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.937817 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:15.938003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.937978 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:15.938102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.938081 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:15.938141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.938122 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plz88\" (UniqueName: \"kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:15.983451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:15.983422 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-0ad26a-serving-cert\"" Nov 02 18:09:16.039655 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.039619 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.039897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.039774 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.039897 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:09:16.039823 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-serving-cert: secret "maas-api-e2e-shared-b-w0-0ad26a-serving-cert" not found Nov 02 18:09:16.039897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.039871 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.040087 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.039901 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plz88\" (UniqueName: \"kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.040087 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:09:16.039939 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls podName:46cb0afe-e0df-46b0-abce-a4dcb62b0fe6 nodeName:}" failed. No retries permitted until 2026-11-02 18:09:16.539914449 +0000 UTC m=+2743.613441634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls") pod "maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" (UID: "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6") : secret "maas-api-e2e-shared-b-w0-0ad26a-serving-cert" not found Nov 02 18:09:16.040665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.040642 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.042555 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.042533 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.099127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.099096 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plz88\" (UniqueName: \"kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.589800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.589760 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.592285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.592257 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:16.797092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:16.796382 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:17.077668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.077637 5647 generic.go:356] "Generic (PLEG): container finished" podID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerID="0c8f31081d6f7f537a4bfa4af71b11c68a84fd89744605289ce12adb643657e5" exitCode=0 Nov 02 18:09:17.082172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.082144 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6211150-1a7a-4f0e-a5bd-0172dea625b1" path="/var/lib/kubelet/pods/f6211150-1a7a-4f0e-a5bd-0172dea625b1/volumes" Nov 02 18:09:17.082492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.082475 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" event={"ID":"9768201e-ace8-4671-8526-64bd24fe8aa5","Type":"ContainerDied","Data":"0c8f31081d6f7f537a4bfa4af71b11c68a84fd89744605289ce12adb643657e5"} Nov 02 18:09:17.082965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.082948 5647 scope.go:122] "RemoveContainer" containerID="0c8f31081d6f7f537a4bfa4af71b11c68a84fd89744605289ce12adb643657e5" Nov 02 18:09:17.084460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.084433 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:09:17.084521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.084498 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:09:17.084566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.084529 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:09:17.174084 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:09:17.174043 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cb0afe_e0df_46b0_abce_a4dcb62b0fe6.slice/crio-74dcb3a40783d3deef6691118331d44806a5a1a665fc737e1ee9e30dac3c211f WatchSource:0}: Error finding container 74dcb3a40783d3deef6691118331d44806a5a1a665fc737e1ee9e30dac3c211f: Status 404 returned error can't find the container with id 74dcb3a40783d3deef6691118331d44806a5a1a665fc737e1ee9e30dac3c211f Nov 02 18:09:17.175538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.175516 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:09:17.191023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:17.190985 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5"] Nov 02 18:09:18.088706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:18.088674 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" event={"ID":"9768201e-ace8-4671-8526-64bd24fe8aa5","Type":"ContainerStarted","Data":"f03a70b29cac96abc0ce0fd773ac7e686517cf51490b1745b27e976f91710b70"} Nov 02 18:09:18.090119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:18.090089 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" event={"ID":"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6","Type":"ContainerStarted","Data":"74dcb3a40783d3deef6691118331d44806a5a1a665fc737e1ee9e30dac3c211f"} Nov 02 18:09:18.220761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:18.216855 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:09:18.385807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:18.385769 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:09:18.387286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:18.387261 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 02 18:09:18.387425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:18.387322 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 02 18:09:19.098093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:19.098048 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" event={"ID":"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6","Type":"ContainerStarted","Data":"07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e"} Nov 02 18:09:19.232015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:19.231960 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" podStartSLOduration=3.342958926 podStartE2EDuration="4.231943333s" totalImagesPullingTime="888.984407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:09:19.228810139 +0000 UTC m=+2746.302337342" watchObservedRunningTime="2026-11-02 18:09:19.231943333 +0000 UTC m=+2746.305470535" Nov 02 18:09:19.399113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:19.397529 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 02 18:09:19.399113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:19.397592 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 02 18:09:20.128803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:20.128771 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:20.385819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:20.385713 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 02 18:09:20.385819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:20.385795 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 02 18:09:21.385980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:21.385940 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 02 18:09:21.386481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:21.386009 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 02 18:09:22.385726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:22.385685 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 02 18:09:22.385983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:22.385787 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 02 18:09:23.386011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:23.385976 5647 patch_prober.go:29] interesting pod/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body= Nov 02 18:09:23.386459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:23.386045 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" Nov 02 18:09:24.390717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:24.390685 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:09:24.391771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:24.391723 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:09:26.189466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:26.189434 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:09:35.948630 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:09:35.948586 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:09:36.524780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:36.524724 5647 scope.go:122] "RemoveContainer" containerID="228ffde832da1e310208a6c253f9d282d6e877548848b21f692c0f561c8b713e" Nov 02 18:09:41.049134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:41.049095 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4"] Nov 02 18:09:41.099537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:41.099504 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef2e72bb-lx7s4"] Nov 02 18:09:42.633680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:42.633642 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:09:43.103580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:43.103542 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e" path="/var/lib/kubelet/pods/cc8529d7-ea1e-4f32-b3b2-49b2bea8a99e/volumes" Nov 02 18:09:47.046668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:47.046628 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5"] Nov 02 18:09:47.097396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:47.097358 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768a2388-jw9j5"] Nov 02 18:09:49.082227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:49.082192 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ad07d5-2919-497e-a811-5f3c9da8c84c" path="/var/lib/kubelet/pods/62ad07d5-2919-497e-a811-5f3c9da8c84c/volumes" Nov 02 18:09:53.429348 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:53.429251 5647 generic.go:356] "Generic (PLEG): container finished" podID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerID="c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a" exitCode=0 Nov 02 18:09:53.429348 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:53.429329 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" event={"ID":"007f9dc9-1629-4cc5-9f79-5355b5c7f11f","Type":"ContainerDied","Data":"c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a"} Nov 02 18:09:53.429869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:53.429846 5647 scope.go:122] "RemoveContainer" containerID="c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a" Nov 02 18:09:53.431557 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:53.431526 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:09:53.431630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:53.431609 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:09:53.431671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:53.431650 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:09:54.438980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:54.438945 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" event={"ID":"007f9dc9-1629-4cc5-9f79-5355b5c7f11f","Type":"ContainerStarted","Data":"bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6"} Nov 02 18:09:54.982264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:54.982221 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:09:54.983799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:54.983729 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 18:09:54.983952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:54.983825 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 18:09:55.981667 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:55.981628 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 18:09:55.982104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:55.981746 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 18:09:56.982350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:56.982310 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 18:09:56.982952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:56.982403 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 18:09:57.981727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:57.981692 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 18:09:57.981939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:57.981778 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 18:09:58.449159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:58.449122 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:09:58.986192 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:58.986155 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 02 18:09:58.986507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:58.986473 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 02 18:09:59.986970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:59.986942 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:09:59.988112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:09:59.988089 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:10:02.050532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:02.050498 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m"] Nov 02 18:10:02.099860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:02.099810 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e38ad7-rjq6m"] Nov 02 18:10:03.081815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:03.081785 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc973c05-8111-4239-a679-8e2ed571d3e9" path="/var/lib/kubelet/pods/cc973c05-8111-4239-a679-8e2ed571d3e9/volumes" Nov 02 18:10:04.404608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.404566 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn"] Nov 02 18:10:04.410099 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.410077 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.525125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.525092 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-de85ba-openshift-default-dockercfg-rnh69\"" Nov 02 18:10:04.614759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.582757 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.628758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.618828 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49qr\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.629124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.629093 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.639818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.629332 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.659811 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.640179 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.719647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.719445 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.744546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.742915 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.744546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.742968 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.744546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.743062 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.746229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.746205 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn"] Nov 02 18:10:04.844998 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.844957 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845009 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49qr\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845037 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845081 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845147 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845178 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845296 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845326 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845605 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845769 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845878 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845855 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.845961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845904 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.846020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.845964 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.847479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.847459 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.847996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.847962 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.903331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.903294 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:04.903489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:04.903352 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49qr\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr\") pod \"e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:05.025554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.025434 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:05.465590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.465560 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn"] Nov 02 18:10:05.465940 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:10:05.465678 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577517bd_d574_4eb8_af37_9c7092c7d59d.slice/crio-014f6093bebcd2544c8e0b1ea22e32b52baad361e0b70173221046721fa1ce2a WatchSource:0}: Error finding container 014f6093bebcd2544c8e0b1ea22e32b52baad361e0b70173221046721fa1ce2a: Status 404 returned error can't find the container with id 014f6093bebcd2544c8e0b1ea22e32b52baad361e0b70173221046721fa1ce2a Nov 02 18:10:05.468437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.468402 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:10:05.468518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.468469 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:10:05.468518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.468498 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:10:05.755868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.755770 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" event={"ID":"577517bd-d574-4eb8-af37-9c7092c7d59d","Type":"ContainerStarted","Data":"45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b"} Nov 02 18:10:05.755868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.755835 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" event={"ID":"577517bd-d574-4eb8-af37-9c7092c7d59d","Type":"ContainerStarted","Data":"014f6093bebcd2544c8e0b1ea22e32b52baad361e0b70173221046721fa1ce2a"} Nov 02 18:10:05.876559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:05.876497 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podStartSLOduration=1.876479304 podStartE2EDuration="1.876479304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:10:05.873754945 +0000 UTC m=+2792.947282149" watchObservedRunningTime="2026-11-02 18:10:05.876479304 +0000 UTC m=+2792.950006508" Nov 02 18:10:05.949069 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:10:05.949025 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:10:06.027216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:06.027086 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:06.056929 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:06.056609 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:06.762287 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:06.762260 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:06.763339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:06.763321 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:10:10.725448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:10.725406 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:10:25.053312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:25.053227 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59"] Nov 02 18:10:25.103861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:25.103827 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d53c5e4c-2cj59"] Nov 02 18:10:27.081986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:27.081948 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00119482-26d5-4ebb-a3b7-6942e73fa5cb" path="/var/lib/kubelet/pods/00119482-26d5-4ebb-a3b7-6942e73fa5cb/volumes" Nov 02 18:10:35.947890 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:10:35.947846 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:10:36.551320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:36.551290 5647 scope.go:122] "RemoveContainer" containerID="c60eb88b9a855679b418aa18db7e5d533e12779ffafefa8275a06270ea985aff" Nov 02 18:10:38.047026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:38.046984 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2"] Nov 02 18:10:38.097212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:38.097177 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27e10ec1-lzbv2"] Nov 02 18:10:39.051323 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:39.051282 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg"] Nov 02 18:10:39.085987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:39.085951 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793def51-6fec-4b5a-98d4-a4b0c33067a8" path="/var/lib/kubelet/pods/793def51-6fec-4b5a-98d4-a4b0c33067a8/volumes" Nov 02 18:10:39.102936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:39.102900 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb342c7-ntrmg"] Nov 02 18:10:41.094633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:41.094600 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0bdbc45-6cde-4466-b68a-4dcb6721c4d6" path="/var/lib/kubelet/pods/a0bdbc45-6cde-4466-b68a-4dcb6721c4d6/volumes" Nov 02 18:10:46.795612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:46.795574 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:10:53.047640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:53.047602 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865"] Nov 02 18:10:53.098470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:53.098438 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865"] Nov 02 18:10:53.130323 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:10:53.130282 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-cddd2ba8-5b865\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="4ba20f5a-781e-4204-ac0c-3f02c509f918" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddd2ba8-5b865" Nov 02 18:10:55.082451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:10:55.082412 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba20f5a-781e-4204-ac0c-3f02c509f918" path="/var/lib/kubelet/pods/4ba20f5a-781e-4204-ac0c-3f02c509f918/volumes" Nov 02 18:11:03.047927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:03.047890 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm"] Nov 02 18:11:03.100401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:03.100354 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5"] Nov 02 18:11:03.117852 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:03.117809 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm"] Nov 02 18:11:03.129783 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:11:03.129698 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-87442f3b-svxtm\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87442f3b-svxtm" Nov 02 18:11:03.131193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:03.131166 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f5f1fcb-h4gf5"] Nov 02 18:11:05.082649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:05.082614 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b790370-dece-408f-9137-69f4b41fe345" path="/var/lib/kubelet/pods/2b790370-dece-408f-9137-69f4b41fe345/volumes" Nov 02 18:11:05.083153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:05.083132 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b3409e-72bc-4c58-a7fc-783dd9c8cf9d" path="/var/lib/kubelet/pods/77b3409e-72bc-4c58-a7fc-783dd9c8cf9d/volumes" Nov 02 18:11:05.949087 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:11:05.949040 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:11:07.280320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:07.280287 5647 generic.go:356] "Generic (PLEG): container finished" podID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerID="4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a" exitCode=0 Nov 02 18:11:07.280770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:07.280329 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" event={"ID":"0324d641-3db4-4347-93ff-f68b9fb6f913","Type":"ContainerDied","Data":"4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a"} Nov 02 18:11:07.280770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:07.280719 5647 scope.go:122] "RemoveContainer" containerID="4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a" Nov 02 18:11:07.282457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:07.282423 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:11:07.282576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:07.282514 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:11:07.282576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:07.282564 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:11:08.291253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:08.291214 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" event={"ID":"0324d641-3db4-4347-93ff-f68b9fb6f913","Type":"ContainerStarted","Data":"04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e"} Nov 02 18:11:08.320290 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:08.320189 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:11:08.320644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:08.320619 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 02 18:11:08.320857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:08.320828 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 02 18:11:09.321183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:09.321143 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 02 18:11:09.321852 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:09.321815 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 02 18:11:10.322765 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:10.318191 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 02 18:11:10.322765 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:10.318255 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 02 18:11:11.326766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:11.325842 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 02 18:11:11.326766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:11.325904 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 02 18:11:12.323174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:12.323131 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 02 18:11:12.323505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:12.323474 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 02 18:11:13.316461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:13.316422 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Nov 02 18:11:13.316920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:13.316493 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Nov 02 18:11:14.321457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:14.321427 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:11:14.321986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:14.321777 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:11:14.322535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:14.322514 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:11:15.046262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:15.046230 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl"] Nov 02 18:11:15.052613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:15.052571 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64a8c37c-cb9sl"] Nov 02 18:11:15.081887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:15.081848 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a187343-1ee4-4bea-8672-f35027e1b89d" path="/var/lib/kubelet/pods/2a187343-1ee4-4bea-8672-f35027e1b89d/volumes" Nov 02 18:11:17.049618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:17.049583 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm"] Nov 02 18:11:17.099860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:17.099826 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29da9c56-t6fbm"] Nov 02 18:11:19.082135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:19.082093 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f79ccba-dfd1-4b0c-9267-9c997de03c00" path="/var/lib/kubelet/pods/1f79ccba-dfd1-4b0c-9267-9c997de03c00/volumes" Nov 02 18:11:22.544658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.544625 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf"] Nov 02 18:11:22.550403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.550379 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.606134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.606099 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf"] Nov 02 18:11:22.644308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.644275 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47hph\" (UniqueName: \"kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.644496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.644318 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.644496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.644440 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.644496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.644477 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.652189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.652163 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-de85ba-serving-cert\"" Nov 02 18:11:22.745613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.745577 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47hph\" (UniqueName: \"kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.745808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.745633 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.745808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.745693 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.745808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.745718 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.745990 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:11:22.745817 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-serving-cert: secret "maas-api-e2e-praxis-b-de85ba-serving-cert" not found Nov 02 18:11:22.745990 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:11:22.745903 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls podName:a0339559-7c20-454e-a219-d05c3399a603 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.245885084 +0000 UTC m=+2870.319412265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls") pod "maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" (UID: "a0339559-7c20-454e-a219-d05c3399a603") : secret "maas-api-e2e-praxis-b-de85ba-serving-cert" not found Nov 02 18:11:22.746455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.746429 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.748321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.748302 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:22.804253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:22.804175 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47hph\" (UniqueName: \"kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:23.256245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:23.256192 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:23.258949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:23.258919 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\") pod \"maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:23.465041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:23.464999 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:23.846491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:23.846463 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf"] Nov 02 18:11:23.848288 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:11:23.848257 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0339559_7c20_454e_a219_d05c3399a603.slice/crio-89f29602417d5c4fd97cfa545ef1368cdb69dca872eea381add1f051d006b346 WatchSource:0}: Error finding container 89f29602417d5c4fd97cfa545ef1368cdb69dca872eea381add1f051d006b346: Status 404 returned error can't find the container with id 89f29602417d5c4fd97cfa545ef1368cdb69dca872eea381add1f051d006b346 Nov 02 18:11:24.427677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:24.427640 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" event={"ID":"a0339559-7c20-454e-a219-d05c3399a603","Type":"ContainerStarted","Data":"89f29602417d5c4fd97cfa545ef1368cdb69dca872eea381add1f051d006b346"} Nov 02 18:11:25.435325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:25.435283 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" event={"ID":"a0339559-7c20-454e-a219-d05c3399a603","Type":"ContainerStarted","Data":"246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051"} Nov 02 18:11:25.435804 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:25.435504 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:31.448622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:31.448587 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:11:31.566245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:31.566184 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" podStartSLOduration=8.641311721 podStartE2EDuration="9.566167972s" totalImagesPullingTime="924.856251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:25.560009926 +0000 UTC m=+2872.633537130" watchObservedRunningTime="2026-11-02 18:11:31.566167972 +0000 UTC m=+2878.639695175" Nov 02 18:11:35.948668 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:11:35.948626 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:11:36.578148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:36.578110 5647 scope.go:122] "RemoveContainer" containerID="fc55e047d0a68682d686902c49224e4e1aaac3052105133960e56f406609dc99" Nov 02 18:11:36.589316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:36.589289 5647 scope.go:122] "RemoveContainer" containerID="5e8aba4e9dcbae9398a52efe519f31ae2d74317c29dd25d55a0d038068e214a9" Nov 02 18:11:36.598968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:36.598948 5647 scope.go:122] "RemoveContainer" containerID="2a803b2ef96cc84e5db6e25f76589d49a0b8832a3750874de8fd03e7234d96f9" Nov 02 18:11:36.609540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:36.609518 5647 scope.go:122] "RemoveContainer" containerID="43a5259a1f091a752a79dcd2b65de0aa80358ea05038f693d56cdac5f1f6300b" Nov 02 18:11:36.618916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:36.618896 5647 scope.go:122] "RemoveContainer" containerID="fca0242590a68cb816f7b1ab7a118263429ee8a32c51ddba302e716c5178a2f1" Nov 02 18:11:36.628252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:36.628225 5647 scope.go:122] "RemoveContainer" containerID="6f09f2155d0b1aa3d9dcd3a6b353d2d054ed9f158af3958221c776e5d2dadda7" Nov 02 18:11:37.809944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:37.809898 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:11:48.687838 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:48.687806 5647 generic.go:356] "Generic (PLEG): container finished" podID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerID="45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b" exitCode=0 Nov 02 18:11:48.688294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:48.687874 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" event={"ID":"577517bd-d574-4eb8-af37-9c7092c7d59d","Type":"ContainerDied","Data":"45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b"} Nov 02 18:11:48.688341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:48.688313 5647 scope.go:122] "RemoveContainer" containerID="45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b" Nov 02 18:11:48.690036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:48.690006 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:11:48.690122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:48.690080 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:11:48.690122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:48.690116 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:11:49.697031 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:49.696994 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" event={"ID":"577517bd-d574-4eb8-af37-9c7092c7d59d","Type":"ContainerStarted","Data":"8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55"} Nov 02 18:11:50.027027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:50.026925 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:11:50.028361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:50.028333 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:11:50.028493 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:50.028400 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 02 18:11:51.027202 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:51.027160 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:11:51.027801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:51.027231 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 02 18:11:51.763030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:51.762997 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:11:52.027234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:52.027140 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:11:52.027234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:52.027218 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 02 18:11:53.027192 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:53.027154 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:11:53.027426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:53.027216 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 02 18:11:54.028484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:54.027936 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:11:54.028484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:54.028010 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 02 18:11:55.026085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:55.026047 5647 patch_prober.go:29] interesting pod/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Nov 02 18:11:55.026278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:55.026120 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Nov 02 18:11:56.030666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:56.030631 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:11:56.032052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:11:56.032029 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:12:05.949015 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:05.948966 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:12:09.021139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.021096 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47"] Nov 02 18:12:09.021570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.021350 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" containerID="cri-o://e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487" gracePeriod=30 Nov 02 18:12:09.071920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.071876 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58"] Nov 02 18:12:09.072144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.072120 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" containerID="cri-o://0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b" gracePeriod=30 Nov 02 18:12:09.081640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.081614 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q"] Nov 02 18:12:09.081885 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.081861 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" containerID="cri-o://b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282" gracePeriod=30 Nov 02 18:12:09.122711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.122676 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86"] Nov 02 18:12:09.122960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.122933 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" containerID="cri-o://ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0" gracePeriod=30 Nov 02 18:12:09.357770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:09.356688 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:12:12.691724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:12.691679 5647 patch_prober.go:29] interesting pod/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:12:12.692283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:12.691781 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:12:14.386918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.386889 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:12:14.418081 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.418025 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 18:12:14.466901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.466857 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.466901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.466909 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.466948 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.466979 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467066 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467181 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467231 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467276 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467300 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467344 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467493 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467528 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467558 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467607 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467658 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467700 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467749 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket\") pod \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\" (UID: \"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9\") " Nov 02 18:12:14.467776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.467775 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket\") pod \"87e626ae-6f46-4428-a1e4-b97129721f21\" (UID: \"87e626ae-6f46-4428-a1e4-b97129721f21\") " Nov 02 18:12:14.468706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.468073 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.468706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.468583 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.468706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.468656 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.468904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.468867 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.468904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.468880 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.469848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.469586 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.469848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.469631 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.470100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.470067 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.470171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.470116 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:14.470785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.470755 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:14.470921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.470905 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.476070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.474134 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:14.476070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.474300 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.476070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.474426 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "kube-api-access-lfr46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.476070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.474434 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:14.476070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.474578 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.476070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.475154 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy" pod "87e626ae-6f46-4428-a1e4-b97129721f21" (UID: "87e626ae-6f46-4428-a1e4-b97129721f21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.476524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.476366 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "kube-api-access-rj9rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.476524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.476475 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy" pod "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" (UID: "f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.487811 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.487785 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:12:14.490118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.490097 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:12:14.575513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.575457 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.576393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.576295 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.576691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.576670 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.576874 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.576305 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.576993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.576486 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.577100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.576857 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577242 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577200 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577242 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577223 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:14.577372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577254 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.577372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577309 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577349 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.577545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577379 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.577545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577442 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577478 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577503 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577550 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.577815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577583 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577637 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.577815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577692 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.577815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577719 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket\") pod \"bb2af564-d05a-412a-bea6-fc57f6b06638\" (UID: \"bb2af564-d05a-412a-bea6-fc57f6b06638\") " Nov 02 18:12:14.577815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.577786 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token\") pod \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\" (UID: \"c6928b40-6f6a-4f4f-890b-4acccb138cc6\") " Nov 02 18:12:14.578248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578225 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.578558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578536 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578567 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj9rb\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-kube-api-access-rj9rb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578583 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578596 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578610 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578623 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578636 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578650 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e626ae-6f46-4428-a1e4-b97129721f21-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.578666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578664 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578677 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e626ae-6f46-4428-a1e4-b97129721f21-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578691 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578705 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578719 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578790 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578907 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578936 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578949 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578963 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578977 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.578991 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e626ae-6f46-4428-a1e4-b97129721f21-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.579351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.579005 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfr46\" (UniqueName: \"kubernetes.io/projected/87e626ae-6f46-4428-a1e4-b97129721f21-kube-api-access-lfr46\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.580246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.579621 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.580246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.579966 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.580246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.580199 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.580246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.580235 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "kube-api-access-xwzgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.580475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.580389 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.581124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.581092 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.581269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.581246 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.582055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.582033 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:14.582372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.582259 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:14.582466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.582406 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.584040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.584014 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "kube-api-access-dg9dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.586160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.586131 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:14.586260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.586176 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token" pod "bb2af564-d05a-412a-bea6-fc57f6b06638" (UID: "bb2af564-d05a-412a-bea6-fc57f6b06638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:14.586260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.586213 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy" pod "c6928b40-6f6a-4f4f-890b-4acccb138cc6" (UID: "c6928b40-6f6a-4f4f-890b-4acccb138cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:14.680806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680768 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9dj\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-kube-api-access-dg9dj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.680806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680811 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.680806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680825 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680839 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680853 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680865 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680877 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680889 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680901 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2af564-d05a-412a-bea6-fc57f6b06638-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680913 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680926 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2af564-d05a-412a-bea6-fc57f6b06638-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680938 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwzgq\" (UniqueName: \"kubernetes.io/projected/c6928b40-6f6a-4f4f-890b-4acccb138cc6-kube-api-access-xwzgq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680952 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2af564-d05a-412a-bea6-fc57f6b06638-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.681100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.680965 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6928b40-6f6a-4f4f-890b-4acccb138cc6-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:14.889404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.889357 5647 generic.go:356] "Generic (PLEG): container finished" podID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerID="b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282" exitCode=0 Nov 02 18:12:14.889669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.889426 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" Nov 02 18:12:14.889669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.889440 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" event={"ID":"bb2af564-d05a-412a-bea6-fc57f6b06638","Type":"ContainerDied","Data":"b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282"} Nov 02 18:12:14.889669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.889523 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q" event={"ID":"bb2af564-d05a-412a-bea6-fc57f6b06638","Type":"ContainerDied","Data":"1cab1637df3911ae326977e062e1f0d2a73399cf8d6e5c864e253b1b714aaaa4"} Nov 02 18:12:14.889669 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.889547 5647 scope.go:122] "RemoveContainer" containerID="b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282" Nov 02 18:12:14.891457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.891429 5647 generic.go:356] "Generic (PLEG): container finished" podID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerID="ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0" exitCode=0 Nov 02 18:12:14.891588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.891521 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" Nov 02 18:12:14.891588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.891563 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" event={"ID":"c6928b40-6f6a-4f4f-890b-4acccb138cc6","Type":"ContainerDied","Data":"ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0"} Nov 02 18:12:14.891720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.891608 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86" event={"ID":"c6928b40-6f6a-4f4f-890b-4acccb138cc6","Type":"ContainerDied","Data":"0fd5ded19a1216644b830952fd966a53a1ba7e13246754f50090d639e04f1b78"} Nov 02 18:12:14.893456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.893429 5647 generic.go:356] "Generic (PLEG): container finished" podID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerID="e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487" exitCode=0 Nov 02 18:12:14.893593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.893493 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" Nov 02 18:12:14.893593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.893495 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" event={"ID":"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9","Type":"ContainerDied","Data":"e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487"} Nov 02 18:12:14.893593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.893562 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47" event={"ID":"f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9","Type":"ContainerDied","Data":"fb258122168dc968b17c26065c5568bc36d34d9fa813fa1c9395153252af0c46"} Nov 02 18:12:14.895846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.895818 5647 generic.go:356] "Generic (PLEG): container finished" podID="87e626ae-6f46-4428-a1e4-b97129721f21" containerID="0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b" exitCode=0 Nov 02 18:12:14.896006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.895868 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" event={"ID":"87e626ae-6f46-4428-a1e4-b97129721f21","Type":"ContainerDied","Data":"0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b"} Nov 02 18:12:14.896006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.895940 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" event={"ID":"87e626ae-6f46-4428-a1e4-b97129721f21","Type":"ContainerDied","Data":"538f7ee8f694730fc9a80232719e05708fe6f070267061413d72ae29dd9b3754"} Nov 02 18:12:14.896006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.895945 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58" Nov 02 18:12:14.912292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.912036 5647 scope.go:122] "RemoveContainer" containerID="b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282" Nov 02 18:12:14.913286 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:14.912454 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282\": container with ID starting with b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282 not found: ID does not exist" containerID="b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282" Nov 02 18:12:14.913286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.912496 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282"} err="failed to get container status \"b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282\": rpc error: code = NotFound desc = could not find container \"b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282\": container with ID starting with b188197bb6dbdf71a1e6d5cfaba088b7d15cfe52e077cc8f4548e38a20204282 not found: ID does not exist" Nov 02 18:12:14.913286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.912528 5647 scope.go:122] "RemoveContainer" containerID="ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0" Nov 02 18:12:14.927768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.927722 5647 scope.go:122] "RemoveContainer" containerID="ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0" Nov 02 18:12:14.928170 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:14.928136 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0\": container with ID starting with ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0 not found: ID does not exist" containerID="ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0" Nov 02 18:12:14.928285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.928181 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0"} err="failed to get container status \"ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0\": rpc error: code = NotFound desc = could not find container \"ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0\": container with ID starting with ada281d2336bc1bbb59e8c84d9d02b7c15847a388ebe5aad3c6abdc38bf46ea0 not found: ID does not exist" Nov 02 18:12:14.928285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.928210 5647 scope.go:122] "RemoveContainer" containerID="e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487" Nov 02 18:12:14.938797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.938771 5647 scope.go:122] "RemoveContainer" containerID="e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487" Nov 02 18:12:14.939148 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:14.939124 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487\": container with ID starting with e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487 not found: ID does not exist" containerID="e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487" Nov 02 18:12:14.939201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.939160 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487"} err="failed to get container status \"e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487\": rpc error: code = NotFound desc = could not find container \"e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487\": container with ID starting with e8645b7e1e2d88abc3a6619427bcdc56921421e2f92d7d46b22bfdb815bc3487 not found: ID does not exist" Nov 02 18:12:14.939201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.939180 5647 scope.go:122] "RemoveContainer" containerID="0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b" Nov 02 18:12:14.952701 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.952672 5647 scope.go:122] "RemoveContainer" containerID="d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11" Nov 02 18:12:14.962933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.962892 5647 scope.go:122] "RemoveContainer" containerID="0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b" Nov 02 18:12:14.963247 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:14.963225 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b\": container with ID starting with 0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b not found: ID does not exist" containerID="0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b" Nov 02 18:12:14.963299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.963256 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b"} err="failed to get container status \"0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b\": rpc error: code = NotFound desc = could not find container \"0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b\": container with ID starting with 0ae84dd40c3543e469399875364a69a8bdf74b34ad20620f2a81b9032913097b not found: ID does not exist" Nov 02 18:12:14.963299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.963275 5647 scope.go:122] "RemoveContainer" containerID="d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11" Nov 02 18:12:14.963560 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:14.963544 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11\": container with ID starting with d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11 not found: ID does not exist" containerID="d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11" Nov 02 18:12:14.963607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:14.963564 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11"} err="failed to get container status \"d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11\": rpc error: code = NotFound desc = could not find container \"d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11\": container with ID starting with d1c90178fb617bd20dd88e15137de2505884e33e5f64e3946a433d05a43dcd11 not found: ID does not exist" Nov 02 18:12:15.237862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.237767 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47"] Nov 02 18:12:15.242933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.242906 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-68r47"] Nov 02 18:12:15.355653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.355621 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58"] Nov 02 18:12:15.358183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.358157 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-vdw58"] Nov 02 18:12:15.470082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.470053 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86"] Nov 02 18:12:15.476567 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.476537 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-g5p86"] Nov 02 18:12:15.589881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.589850 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q"] Nov 02 18:12:15.596894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:15.596863 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ceb1ae-openshift-default-55d9647b55-fwx2q"] Nov 02 18:12:17.082176 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:17.082145 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" path="/var/lib/kubelet/pods/87e626ae-6f46-4428-a1e4-b97129721f21/volumes" Nov 02 18:12:17.082662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:17.082649 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" path="/var/lib/kubelet/pods/bb2af564-d05a-412a-bea6-fc57f6b06638/volumes" Nov 02 18:12:17.083130 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:17.083116 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" path="/var/lib/kubelet/pods/c6928b40-6f6a-4f4f-890b-4acccb138cc6/volumes" Nov 02 18:12:17.083863 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:17.083849 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" path="/var/lib/kubelet/pods/f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9/volumes" Nov 02 18:12:21.108339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:21.108302 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm"] Nov 02 18:12:21.108831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:21.108539 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" containerID="cri-o://799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528" gracePeriod=30 Nov 02 18:12:26.387941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.387829 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:12:26.538856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.538824 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.538902 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539062 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539068 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:26.539135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539096 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539160 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539218 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539314 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539355 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539374 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:26.539408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539383 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket\") pod \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\" (UID: \"fa77dbd2-aec7-4e43-93ee-61772d5e6953\") " Nov 02 18:12:26.539712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539396 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:12:26.539712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539653 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:26.539942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.539669 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:26.540310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.540286 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.540399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.540315 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.540399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.540333 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.540399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.540351 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.540399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.540367 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.541529 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.541492 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:12:26.541630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.541532 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:12:26.541630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.541582 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "kube-api-access-w8msp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:26.541708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.541690 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token" pod "fa77dbd2-aec7-4e43-93ee-61772d5e6953" (UID: "fa77dbd2-aec7-4e43-93ee-61772d5e6953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:12:26.641570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.641470 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8msp\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-kube-api-access-w8msp\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.641570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.641505 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.641570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.641520 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.641570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.641535 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa77dbd2-aec7-4e43-93ee-61772d5e6953-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:12:26.982704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.982619 5647 generic.go:356] "Generic (PLEG): container finished" podID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerID="799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528" exitCode=0 Nov 02 18:12:26.982704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.982680 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" event={"ID":"fa77dbd2-aec7-4e43-93ee-61772d5e6953","Type":"ContainerDied","Data":"799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528"} Nov 02 18:12:26.982704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.982686 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" Nov 02 18:12:26.983043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.982721 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm" event={"ID":"fa77dbd2-aec7-4e43-93ee-61772d5e6953","Type":"ContainerDied","Data":"85c3f9c1623bbeff44269391cda29824d5e7d12030ffd39920dfe6a0f29cf1f9"} Nov 02 18:12:26.983043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.982764 5647 scope.go:122] "RemoveContainer" containerID="799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528" Nov 02 18:12:26.993815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.993796 5647 scope.go:122] "RemoveContainer" containerID="799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528" Nov 02 18:12:26.994101 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:26.994080 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528\": container with ID starting with 799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528 not found: ID does not exist" containerID="799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528" Nov 02 18:12:26.994179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:26.994110 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528"} err="failed to get container status \"799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528\": rpc error: code = NotFound desc = could not find container \"799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528\": container with ID starting with 799c541077f0ea0654bb065f2f68359ce401d83dbdd54833c2cb0157dba81528 not found: ID does not exist" Nov 02 18:12:27.155288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:27.155253 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm"] Nov 02 18:12:27.205712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:27.205681 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-2mjrm"] Nov 02 18:12:29.083258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:29.083212 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" path="/var/lib/kubelet/pods/fa77dbd2-aec7-4e43-93ee-61772d5e6953/volumes" Nov 02 18:12:35.949601 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:12:35.949562 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:12:36.680760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:36.680695 5647 scope.go:122] "RemoveContainer" containerID="ad1bc39271de11e75f816715b908c0e3b96edf9e73455f1b5c6ba47cdf721051" Nov 02 18:12:36.690705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:36.690681 5647 scope.go:122] "RemoveContainer" containerID="6fc3570136c3d9db8ed39e5cfd21444ea982919bb0a24c7c431171344dcb06d6" Nov 02 18:12:36.699865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:36.699842 5647 scope.go:122] "RemoveContainer" containerID="dc1bfd5ed8537e5b5edcf998ce370827dece9b94dbc0a1c8550a52f951550f98" Nov 02 18:12:36.708989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:36.708966 5647 scope.go:122] "RemoveContainer" containerID="3ce4bcb2a2a224db897d450b176bc39ae8bc87cc9fe1e46fafce081c648f96cf" Nov 02 18:12:36.718477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:36.718456 5647 scope.go:122] "RemoveContainer" containerID="c577f0ef46d69add00b3c9bcd37629fcf39c340ae2f0157dd6cfb0cba0431c42" Nov 02 18:12:40.220846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.220806 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg"] Nov 02 18:12:40.221560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221540 5647 cpu_manager.go:414] "containerMap: removing container" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221563 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221582 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221587 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221605 5647 cpu_manager.go:414] "containerMap: removing container" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221610 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221625 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" Nov 02 18:12:40.221631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221629 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221637 5647 cpu_manager.go:414] "containerMap: removing container" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221642 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221657 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221662 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221792 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa77dbd2-aec7-4e43-93ee-61772d5e6953" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221808 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f814ce19-52ea-4bcc-ba18-ac5bd4b2e0c9" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221816 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221823 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6928b40-6f6a-4f4f-890b-4acccb138cc6" containerName="istio-proxy" Nov 02 18:12:40.221912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.221831 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2af564-d05a-412a-bea6-fc57f6b06638" containerName="istio-proxy" Nov 02 18:12:40.227057 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.227037 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.282341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.282305 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg"] Nov 02 18:12:40.329180 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.329148 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"openshift-service-ca.crt\"" Nov 02 18:12:40.329180 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.329164 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"kube-root-ca.crt\"" Nov 02 18:12:40.331560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.331540 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"auth-test-model-0ad26a-kserve-self-signed-certs\"" Nov 02 18:12:40.331796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.331766 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-0ad26a\"/\"default-dockercfg-cmd9m\"" Nov 02 18:12:40.343635 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.343607 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.343774 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.343659 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.343774 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.343760 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.343889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.343871 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.343945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.343933 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.344008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.343995 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4ql6\" (UniqueName: \"kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445089 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445158 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4ql6\" (UniqueName: \"kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445261 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445305 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445328 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445389 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445544 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445660 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.445794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.445774 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.447772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.447746 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.448059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.448042 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.503891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.503815 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4ql6\" (UniqueName: \"kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6\") pod \"auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.539677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.539639 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:40.923958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:40.923930 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg"] Nov 02 18:12:40.924281 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:12:40.924258 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce11cc88_9b1c_4708_822e_d9ecd6ce29c9.slice/crio-1bd4fcdfdf148bc20a1dbe66e540ddc86444419b77609c7973ba8e8e10807641 WatchSource:0}: Error finding container 1bd4fcdfdf148bc20a1dbe66e540ddc86444419b77609c7973ba8e8e10807641: Status 404 returned error can't find the container with id 1bd4fcdfdf148bc20a1dbe66e540ddc86444419b77609c7973ba8e8e10807641 Nov 02 18:12:41.091661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:41.091628 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" event={"ID":"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9","Type":"ContainerStarted","Data":"bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7"} Nov 02 18:12:41.091859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:41.091674 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" event={"ID":"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9","Type":"ContainerStarted","Data":"1bd4fcdfdf148bc20a1dbe66e540ddc86444419b77609c7973ba8e8e10807641"} Nov 02 18:12:41.208064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:41.206488 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" podStartSLOduration=1.206469965 podStartE2EDuration="1.206469965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:41.204300091 +0000 UTC m=+2948.277827288" watchObservedRunningTime="2026-11-02 18:12:41.206469965 +0000 UTC m=+2948.279997169" Nov 02 18:12:41.540788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:41.540685 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:50.540900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:50.540858 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:50.554201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:50.554174 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:51.194587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:51.194553 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:12:53.013746 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:53.013693 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:12:55.272960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:12:55.272709 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p"] Nov 02 18:13:00.274049 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.274005 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" podUID="3a757d14-821c-48e7-8c05-beb593e97dc0" containerName="payload-pre-processing" containerID="cri-o://d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55" gracePeriod=25 Nov 02 18:13:00.556993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.556946 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:13:00.563466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.563437 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc\" (UniqueName: \"kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc\") pod \"3a757d14-821c-48e7-8c05-beb593e97dc0\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " Nov 02 18:13:00.563663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.563602 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume\") pod \"3a757d14-821c-48e7-8c05-beb593e97dc0\" (UID: \"3a757d14-821c-48e7-8c05-beb593e97dc0\") " Nov 02 18:13:00.564466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.564434 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume" pod "3a757d14-821c-48e7-8c05-beb593e97dc0" (UID: "3a757d14-821c-48e7-8c05-beb593e97dc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:00.565953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.565919 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc" pod "3a757d14-821c-48e7-8c05-beb593e97dc0" (UID: "3a757d14-821c-48e7-8c05-beb593e97dc0"). InnerVolumeSpecName "kube-api-access-t9pgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:00.664845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.664805 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9pgc\" (UniqueName: \"kubernetes.io/projected/3a757d14-821c-48e7-8c05-beb593e97dc0-kube-api-access-t9pgc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:00.664845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:00.664839 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a757d14-821c-48e7-8c05-beb593e97dc0-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:01.249500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.249462 5647 generic.go:356] "Generic (PLEG): container finished" podID="3a757d14-821c-48e7-8c05-beb593e97dc0" containerID="d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55" exitCode=0 Nov 02 18:13:01.249705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.249515 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" Nov 02 18:13:01.249705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.249555 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" event={"ID":"3a757d14-821c-48e7-8c05-beb593e97dc0","Type":"ContainerDied","Data":"d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55"} Nov 02 18:13:01.249705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.249616 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p" event={"ID":"3a757d14-821c-48e7-8c05-beb593e97dc0","Type":"ContainerDied","Data":"a86cb42f073f3d1b836cddacbea1f1fbca593103362aecb62736129386a798c2"} Nov 02 18:13:01.249705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.249639 5647 scope.go:122] "RemoveContainer" containerID="d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55" Nov 02 18:13:01.262210 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.262195 5647 scope.go:122] "RemoveContainer" containerID="d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55" Nov 02 18:13:01.262694 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:13:01.262662 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55\": container with ID starting with d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55 not found: ID does not exist" containerID="d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55" Nov 02 18:13:01.262836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.262701 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55"} err="failed to get container status \"d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55\": rpc error: code = NotFound desc = could not find container \"d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55\": container with ID starting with d2ececf7273cb052fae20c554dee7424eef1535801c0ee57c090338ba5793a55 not found: ID does not exist" Nov 02 18:13:01.416872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.416778 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p"] Nov 02 18:13:01.467368 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:01.467336 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-eb1633-cf74d55dc-rg72p"] Nov 02 18:13:03.083212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:03.083163 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a757d14-821c-48e7-8c05-beb593e97dc0" path="/var/lib/kubelet/pods/3a757d14-821c-48e7-8c05-beb593e97dc0/volumes" Nov 02 18:13:05.947656 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:13:05.947615 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:13:34.364179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:34.364095 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:13:34.366841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:34.366821 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:13:35.947768 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:13:35.947707 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:13:44.493591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:44.493553 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44"] Nov 02 18:13:44.494057 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:44.493844 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" containerID="cri-o://f03a70b29cac96abc0ce0fd773ac7e686517cf51490b1745b27e976f91710b70" gracePeriod=30 Nov 02 18:13:49.667575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.667541 5647 generic.go:356] "Generic (PLEG): container finished" podID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerID="f03a70b29cac96abc0ce0fd773ac7e686517cf51490b1745b27e976f91710b70" exitCode=0 Nov 02 18:13:49.668117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.667610 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" event={"ID":"9768201e-ace8-4671-8526-64bd24fe8aa5","Type":"ContainerDied","Data":"f03a70b29cac96abc0ce0fd773ac7e686517cf51490b1745b27e976f91710b70"} Nov 02 18:13:49.668117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.667671 5647 scope.go:122] "RemoveContainer" containerID="0c8f31081d6f7f537a4bfa4af71b11c68a84fd89744605289ce12adb643657e5" Nov 02 18:13:49.764937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.764870 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:13:49.853540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853501 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.853722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853554 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.853722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.853722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853680 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.853722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853702 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.853997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853775 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.853997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853818 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:49.853997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853944 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.854184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.853941 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:49.854184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854000 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.854184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854044 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:49.854184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854075 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo\") pod \"9768201e-ace8-4671-8526-64bd24fe8aa5\" (UID: \"9768201e-ace8-4671-8526-64bd24fe8aa5\") " Nov 02 18:13:49.854407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854236 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:49.854407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854255 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:49.854870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854849 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9768201e-ace8-4671-8526-64bd24fe8aa5-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.854946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854877 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.854946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854894 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.854946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854909 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.854946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.854924 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.856162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.856130 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "kube-api-access-ghnj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:49.856162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.856142 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:49.856591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.856573 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:13:49.856642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.856586 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy" pod "9768201e-ace8-4671-8526-64bd24fe8aa5" (UID: "9768201e-ace8-4671-8526-64bd24fe8aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:13:49.956094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.956062 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.956094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.956093 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.956094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.956106 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnj6\" (UniqueName: \"kubernetes.io/projected/9768201e-ace8-4671-8526-64bd24fe8aa5-kube-api-access-ghnj6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:49.956365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:49.956121 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9768201e-ace8-4671-8526-64bd24fe8aa5-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:50.522489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:50.522447 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:13:50.674986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:50.674948 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" event={"ID":"9768201e-ace8-4671-8526-64bd24fe8aa5","Type":"ContainerDied","Data":"c535d39756a6212c4d758d483a429e8a7a6b6a2426a8aef3507254f288160bbf"} Nov 02 18:13:50.674986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:50.674970 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44" Nov 02 18:13:50.675536 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:50.675014 5647 scope.go:122] "RemoveContainer" containerID="f03a70b29cac96abc0ce0fd773ac7e686517cf51490b1745b27e976f91710b70" Nov 02 18:13:50.841455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:50.841418 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44"] Nov 02 18:13:50.892260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:50.892224 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-eb1633-openshift-default-85c64b87df-fbn44"] Nov 02 18:13:51.027921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.027886 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf"] Nov 02 18:13:51.028143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.028118 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" podUID="a0339559-7c20-454e-a219-d05c3399a603" containerName="maas-api" containerID="cri-o://246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051" gracePeriod=30 Nov 02 18:13:51.083907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.083869 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" path="/var/lib/kubelet/pods/9768201e-ace8-4671-8526-64bd24fe8aa5/volumes" Nov 02 18:13:51.362462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.362435 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:13:51.494808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.488249 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs\") pod \"a0339559-7c20-454e-a219-d05c3399a603\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " Nov 02 18:13:51.494808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.488412 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca\") pod \"a0339559-7c20-454e-a219-d05c3399a603\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " Nov 02 18:13:51.494808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.488445 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\") pod \"a0339559-7c20-454e-a219-d05c3399a603\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " Nov 02 18:13:51.494808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.488556 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph\" (UniqueName: \"kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph\") pod \"a0339559-7c20-454e-a219-d05c3399a603\" (UID: \"a0339559-7c20-454e-a219-d05c3399a603\") " Nov 02 18:13:51.494808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.489152 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca" pod "a0339559-7c20-454e-a219-d05c3399a603" (UID: "a0339559-7c20-454e-a219-d05c3399a603"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:13:51.515923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.498082 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0339559-7c20-454e-a219-d05c3399a603-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:51.515923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.504003 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph" pod "a0339559-7c20-454e-a219-d05c3399a603" (UID: "a0339559-7c20-454e-a219-d05c3399a603"). InnerVolumeSpecName "kube-api-access-47hph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:13:51.515923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.504568 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls" pod "a0339559-7c20-454e-a219-d05c3399a603" (UID: "a0339559-7c20-454e-a219-d05c3399a603"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:51.515923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.512830 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs" pod "a0339559-7c20-454e-a219-d05c3399a603" (UID: "a0339559-7c20-454e-a219-d05c3399a603"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:13:51.599712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.599677 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47hph\" (UniqueName: \"kubernetes.io/projected/a0339559-7c20-454e-a219-d05c3399a603-kube-api-access-47hph\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:51.599712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.599711 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:51.599712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.599722 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0339559-7c20-454e-a219-d05c3399a603-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:13:51.685435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.685398 5647 generic.go:356] "Generic (PLEG): container finished" podID="a0339559-7c20-454e-a219-d05c3399a603" containerID="246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051" exitCode=0 Nov 02 18:13:51.685871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.685563 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" event={"ID":"a0339559-7c20-454e-a219-d05c3399a603","Type":"ContainerDied","Data":"246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051"} Nov 02 18:13:51.685871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.685633 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" event={"ID":"a0339559-7c20-454e-a219-d05c3399a603","Type":"ContainerDied","Data":"89f29602417d5c4fd97cfa545ef1368cdb69dca872eea381add1f051d006b346"} Nov 02 18:13:51.685871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.685653 5647 scope.go:122] "RemoveContainer" containerID="246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051" Nov 02 18:13:51.685871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.685794 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf" Nov 02 18:13:51.700292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.700269 5647 scope.go:122] "RemoveContainer" containerID="246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051" Nov 02 18:13:51.700616 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:13:51.700595 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051\": container with ID starting with 246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051 not found: ID does not exist" containerID="246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051" Nov 02 18:13:51.700682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.700623 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051"} err="failed to get container status \"246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051\": rpc error: code = NotFound desc = could not find container \"246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051\": container with ID starting with 246f7ea2c59a0b795ccdd85df2288eb362cebf287ff978c42e2082f3a1de2051 not found: ID does not exist" Nov 02 18:13:51.865790 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.865721 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf"] Nov 02 18:13:51.869721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:51.869687 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-de85ba-6d9658bdcf-s42nf"] Nov 02 18:13:53.098343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:13:53.098308 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0339559-7c20-454e-a219-d05c3399a603" path="/var/lib/kubelet/pods/a0339559-7c20-454e-a219-d05c3399a603/volumes" Nov 02 18:14:04.003748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:04.003699 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn"] Nov 02 18:14:04.004213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:04.003961 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" containerID="cri-o://8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55" gracePeriod=30 Nov 02 18:14:05.395426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.395391 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d"] Nov 02 18:14:05.396055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396016 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3a757d14-821c-48e7-8c05-beb593e97dc0" containerName="payload-pre-processing" Nov 02 18:14:05.396055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396032 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a757d14-821c-48e7-8c05-beb593e97dc0" containerName="payload-pre-processing" Nov 02 18:14:05.396055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396050 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a0339559-7c20-454e-a219-d05c3399a603" containerName="maas-api" Nov 02 18:14:05.396055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396056 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0339559-7c20-454e-a219-d05c3399a603" containerName="maas-api" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396094 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396099 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396110 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396114 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396197 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a757d14-821c-48e7-8c05-beb593e97dc0" containerName="payload-pre-processing" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396207 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0339559-7c20-454e-a219-d05c3399a603" containerName="maas-api" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396217 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" Nov 02 18:14:05.396225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396224 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9768201e-ace8-4671-8526-64bd24fe8aa5" containerName="istio-proxy" Nov 02 18:14:05.396517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.396231 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e626ae-6f46-4428-a1e4-b97129721f21" containerName="istio-proxy" Nov 02 18:14:05.399246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.399230 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426008 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426096 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426165 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426236 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmdj\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426298 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426332 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426397 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426478 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.429757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.426522 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.523821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.523405 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-3dc055-openshift-default-dockercfg-jlwcr\"" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.562516 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.562749 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmdj\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.562917 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.563049 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.563224 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.563522 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.563676 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.563810 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.563886 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.565657 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.565775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.565763 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.595799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.566360 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.596159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.580231 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.596316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.580565 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.626190 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.625952 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.626419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.626332 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.632044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.632014 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.632388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.632360 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmdj\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj\") pod \"e2e-explicit-3dc055-openshift-default-846886d5-n5k5d\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.635495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.635470 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d"] Nov 02 18:14:05.715360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:05.715256 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:05.948810 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:14:05.948769 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:14:06.182969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.182940 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d"] Nov 02 18:14:06.184591 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:14:06.184560 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289c4ea7_7005_4b14_92c3_f963d3a7242e.slice/crio-dac81dbce77b1c38c3d7a0d933e601680da7c699133e446ae7dca18b5041089c WatchSource:0}: Error finding container dac81dbce77b1c38c3d7a0d933e601680da7c699133e446ae7dca18b5041089c: Status 404 returned error can't find the container with id dac81dbce77b1c38c3d7a0d933e601680da7c699133e446ae7dca18b5041089c Nov 02 18:14:06.187582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.187541 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:14:06.187685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.187629 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:14:06.187685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.187670 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:14:06.787806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.787768 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" event={"ID":"289c4ea7-7005-4b14-92c3-f963d3a7242e","Type":"ContainerStarted","Data":"8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029"} Nov 02 18:14:06.788312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.787823 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" event={"ID":"289c4ea7-7005-4b14-92c3-f963d3a7242e","Type":"ContainerStarted","Data":"dac81dbce77b1c38c3d7a0d933e601680da7c699133e446ae7dca18b5041089c"} Nov 02 18:14:06.908075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:06.908021 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podStartSLOduration=1.908006625 podStartE2EDuration="1.908006625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:06.906864597 +0000 UTC m=+3033.980391801" watchObservedRunningTime="2026-11-02 18:14:06.908006625 +0000 UTC m=+3033.981533827" Nov 02 18:14:07.718670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:07.718631 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:07.724490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:07.724463 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:07.794976 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:07.794944 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:07.795784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:07.795766 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:14:09.269028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.268967 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:14:09.414063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414028 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414069 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414128 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414145 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414172 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414287 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414325 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414358 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:09.414609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414376 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414592 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:09.414815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414608 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket\") pod \"577517bd-d574-4eb8-af37-9c7092c7d59d\" (UID: \"577517bd-d574-4eb8-af37-9c7092c7d59d\") " Nov 02 18:14:09.414815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414712 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:14:09.414815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414768 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:09.414972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.414902 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:09.415464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.415435 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.415464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.415465 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.415688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.415481 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/577517bd-d574-4eb8-af37-9c7092c7d59d-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.415688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.415497 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.415688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.415511 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.416757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.416701 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:14:09.416889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.416768 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "kube-api-access-v49qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:09.416889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.416812 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:14:09.416987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.416900 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy" pod "577517bd-d574-4eb8-af37-9c7092c7d59d" (UID: "577517bd-d574-4eb8-af37-9c7092c7d59d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:14:09.516721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.516686 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49qr\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-kube-api-access-v49qr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.516721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.516720 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.516968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.516751 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.516968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.516763 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/577517bd-d574-4eb8-af37-9c7092c7d59d-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:14:09.809370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.809338 5647 generic.go:356] "Generic (PLEG): container finished" podID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerID="8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55" exitCode=0 Nov 02 18:14:09.809586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.809387 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" Nov 02 18:14:09.809586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.809430 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" event={"ID":"577517bd-d574-4eb8-af37-9c7092c7d59d","Type":"ContainerDied","Data":"8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55"} Nov 02 18:14:09.809586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.809500 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn" event={"ID":"577517bd-d574-4eb8-af37-9c7092c7d59d","Type":"ContainerDied","Data":"014f6093bebcd2544c8e0b1ea22e32b52baad361e0b70173221046721fa1ce2a"} Nov 02 18:14:09.809586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.809533 5647 scope.go:122] "RemoveContainer" containerID="8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55" Nov 02 18:14:09.820718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.820698 5647 scope.go:122] "RemoveContainer" containerID="45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b" Nov 02 18:14:09.829967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.829947 5647 scope.go:122] "RemoveContainer" containerID="8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55" Nov 02 18:14:09.830250 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:14:09.830229 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55\": container with ID starting with 8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55 not found: ID does not exist" containerID="8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55" Nov 02 18:14:09.830309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.830260 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55"} err="failed to get container status \"8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55\": rpc error: code = NotFound desc = could not find container \"8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55\": container with ID starting with 8839cd399e803af890ea7c9a7471785e3f462d76276e6a4d5a9dd5cb03e39e55 not found: ID does not exist" Nov 02 18:14:09.830309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.830278 5647 scope.go:122] "RemoveContainer" containerID="45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b" Nov 02 18:14:09.830525 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:14:09.830507 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b\": container with ID starting with 45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b not found: ID does not exist" containerID="45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b" Nov 02 18:14:09.830579 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:09.830529 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b"} err="failed to get container status \"45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b\": rpc error: code = NotFound desc = could not find container \"45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b\": container with ID starting with 45abb91e7dd29b2652da773d1474c130847a397fa047f544e2593a527567567b not found: ID does not exist" Nov 02 18:14:10.008289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:10.008253 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn"] Nov 02 18:14:10.042656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:10.042621 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-de85ba-openshift-default-7c76f47677-n9nfn"] Nov 02 18:14:11.081932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:11.081888 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" path="/var/lib/kubelet/pods/577517bd-d574-4eb8-af37-9c7092c7d59d/volumes" Nov 02 18:14:13.435703 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.435671 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt"] Nov 02 18:14:13.436294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.436279 5647 cpu_manager.go:414] "containerMap: removing container" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" Nov 02 18:14:13.436294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.436295 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" Nov 02 18:14:13.436389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.436330 5647 cpu_manager.go:414] "containerMap: removing container" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" Nov 02 18:14:13.436389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.436335 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" Nov 02 18:14:13.436455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.436420 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" Nov 02 18:14:13.436455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.436432 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="577517bd-d574-4eb8-af37-9c7092c7d59d" containerName="istio-proxy" Nov 02 18:14:13.441103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.441079 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.500386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.500340 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt"] Nov 02 18:14:13.577748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.577689 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.578090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.578066 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjv4\" (UniqueName: \"kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4\") pod \"payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.588756 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.584991 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3dc055\"" Nov 02 18:14:13.632116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.590913 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3dc055-dockercfg-gt98k\"" Nov 02 18:14:13.636722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.636688 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72"] Nov 02 18:14:13.648122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.648091 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72"] Nov 02 18:14:13.648375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.648310 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.679980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.679941 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.680185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.680062 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjv4\" (UniqueName: \"kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4\") pod \"payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.680660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.680638 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.738286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.738214 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjv4\" (UniqueName: \"kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4\") pod \"payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.754796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.754765 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:13.781611 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.781577 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.781818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.781624 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4n9\" (UniqueName: \"kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9\") pod \"payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.883614 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.883579 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.883857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.883620 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4n9\" (UniqueName: \"kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9\") pod \"payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.884371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.884350 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.941441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.941408 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4n9\" (UniqueName: \"kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9\") pod \"payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:13.972051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:13.972015 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:14.133083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.133052 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt"] Nov 02 18:14:14.133523 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:14:14.133500 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c319ea_ef07_460d_80a1_5713420b9d6f.slice/crio-986f78baa3b00723b280396e956e2ce87bd5bdc60829e4c2fd2989f98a966678 WatchSource:0}: Error finding container 986f78baa3b00723b280396e956e2ce87bd5bdc60829e4c2fd2989f98a966678: Status 404 returned error can't find the container with id 986f78baa3b00723b280396e956e2ce87bd5bdc60829e4c2fd2989f98a966678 Nov 02 18:14:14.347683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.347646 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72"] Nov 02 18:14:14.348686 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:14:14.348657 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b435d0_c9a9_4bd9_97c5_dbc4e40379f0.slice/crio-7f38baf35d40a37601e5f854d38e0192879a2c8a2ec81f4364bccdb3678e54c9 WatchSource:0}: Error finding container 7f38baf35d40a37601e5f854d38e0192879a2c8a2ec81f4364bccdb3678e54c9: Status 404 returned error can't find the container with id 7f38baf35d40a37601e5f854d38e0192879a2c8a2ec81f4364bccdb3678e54c9 Nov 02 18:14:14.845463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.845423 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" event={"ID":"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0","Type":"ContainerStarted","Data":"98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d"} Nov 02 18:14:14.845961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.845518 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" event={"ID":"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0","Type":"ContainerStarted","Data":"7f38baf35d40a37601e5f854d38e0192879a2c8a2ec81f4364bccdb3678e54c9"} Nov 02 18:14:14.845961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.845551 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:14.847141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.847115 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" event={"ID":"47c319ea-ef07-460d-80a1-5713420b9d6f","Type":"ContainerStarted","Data":"44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93"} Nov 02 18:14:14.847310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.847156 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" event={"ID":"47c319ea-ef07-460d-80a1-5713420b9d6f","Type":"ContainerStarted","Data":"986f78baa3b00723b280396e956e2ce87bd5bdc60829e4c2fd2989f98a966678"} Nov 02 18:14:14.847310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.847176 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:14.964340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:14.964291 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" podStartSLOduration=1.96427702 podStartE2EDuration="1.96427702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:14.96269972 +0000 UTC m=+3042.036226923" watchObservedRunningTime="2026-11-02 18:14:14.96427702 +0000 UTC m=+3042.037804238" Nov 02 18:14:25.858457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:25.858427 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:14:25.858962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:25.858497 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:14:25.980002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:25.979948 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" podStartSLOduration=12.9799323 podStartE2EDuration="12.9799323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:15.099170054 +0000 UTC m=+3042.172697254" watchObservedRunningTime="2026-11-02 18:14:25.9799323 +0000 UTC m=+3053.053459500" Nov 02 18:14:31.941554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:31.941517 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:14:35.947549 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:14:35.947510 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:14:47.033692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.033602 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf"] Nov 02 18:14:47.039457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.039437 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.093018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.092983 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-eb709f-openshift-default-dockercfg-j27hk\"" Nov 02 18:14:47.145456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.145419 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.145456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.145469 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.145715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.145617 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.145715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.145647 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ft8\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.145715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.145678 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.148772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.145712 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.148772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.147695 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf"] Nov 02 18:14:47.148772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.147945 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.148772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.148144 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.148772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.148183 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249569 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249710 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249762 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249805 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249838 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249909 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249950 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ft8\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.249990 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.250023 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.251759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.250486 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.266799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.266416 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.269258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.267758 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.269258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.268688 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.269258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.269216 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.342841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.342223 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.372324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.372258 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.375510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.375444 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ft8\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.381082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.381030 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token\") pod \"e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:47.654239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:47.654141 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:48.091352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:48.091324 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf"] Nov 02 18:14:48.092850 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:14:48.092822 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ecb0add_9f6e_478f_a1dd_0e5524c4bf05.slice/crio-88795268101ca5a363aabfdb13d91821a701f0dfc7cf2f831459e7005b47ac1b WatchSource:0}: Error finding container 88795268101ca5a363aabfdb13d91821a701f0dfc7cf2f831459e7005b47ac1b: Status 404 returned error can't find the container with id 88795268101ca5a363aabfdb13d91821a701f0dfc7cf2f831459e7005b47ac1b Nov 02 18:14:48.095383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:48.095349 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:14:48.095478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:48.095419 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:14:48.095478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:48.095449 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:14:49.067074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:49.067033 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" event={"ID":"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05","Type":"ContainerStarted","Data":"1a43bdfc771d0d5d1893f35c7f949a08f68255825657e1208a21e130de09f606"} Nov 02 18:14:49.067074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:49.067080 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" event={"ID":"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05","Type":"ContainerStarted","Data":"88795268101ca5a363aabfdb13d91821a701f0dfc7cf2f831459e7005b47ac1b"} Nov 02 18:14:49.288768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:49.288704 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podStartSLOduration=3.288690634 podStartE2EDuration="3.288690634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:49.235585155 +0000 UTC m=+3076.309112359" watchObservedRunningTime="2026-11-02 18:14:49.288690634 +0000 UTC m=+3076.362217837" Nov 02 18:14:49.654505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:49.654468 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:49.660631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:49.660607 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:50.074451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:50.074367 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:50.075381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:50.075363 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:14:55.996778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:55.993437 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p"] Nov 02 18:14:56.003649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.002965 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.052681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.052651 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p"] Nov 02 18:14:56.081373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.081339 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.081590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.081471 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24c7z\" (UniqueName: \"kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z\") pod \"payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.100809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.100773 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-eb709f\"" Nov 02 18:14:56.100954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.100840 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-eb709f-dockercfg-w48tq\"" Nov 02 18:14:56.182430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.182396 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24c7z\" (UniqueName: \"kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z\") pod \"payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.182637 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.182598 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.183390 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.183364 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.241119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.241089 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24c7z\" (UniqueName: \"kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z\") pod \"payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.346456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.346429 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:56.726515 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:56.726476 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p"] Nov 02 18:14:56.731097 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:14:56.731012 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfd12b6_aaa2_494c_b6c8_9aa0968b87dd.slice/crio-63372cb1b64ceffedccdd39001d7d6fb30d843b134e16ae9ad61c49b2a6ef638 WatchSource:0}: Error finding container 63372cb1b64ceffedccdd39001d7d6fb30d843b134e16ae9ad61c49b2a6ef638: Status 404 returned error can't find the container with id 63372cb1b64ceffedccdd39001d7d6fb30d843b134e16ae9ad61c49b2a6ef638 Nov 02 18:14:57.115952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:57.115915 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" event={"ID":"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd","Type":"ContainerStarted","Data":"9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6"} Nov 02 18:14:57.116369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:57.115975 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:14:57.116369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:57.115999 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" event={"ID":"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd","Type":"ContainerStarted","Data":"63372cb1b64ceffedccdd39001d7d6fb30d843b134e16ae9ad61c49b2a6ef638"} Nov 02 18:14:57.233151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:14:57.233096 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" podStartSLOduration=2.233081499 podStartE2EDuration="2.233081499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:57.231176017 +0000 UTC m=+3084.304703224" watchObservedRunningTime="2026-11-02 18:14:57.233081499 +0000 UTC m=+3084.306608702" Nov 02 18:15:00.171751 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.171706 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l"] Nov 02 18:15:00.175904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.175876 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp"] Nov 02 18:15:00.176084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.176011 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.179994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.179977 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.191397 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.191373 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7"] Nov 02 18:15:00.196054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.196029 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.222704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.222669 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v"] Nov 02 18:15:00.226917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.226896 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2"] Nov 02 18:15:00.227038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.227007 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.230649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.230628 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.239507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.239486 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:15:00.274474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.274439 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l"] Nov 02 18:15:00.278319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.278295 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp"] Nov 02 18:15:00.323682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.323637 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v"] Nov 02 18:15:00.334646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.333833 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7"] Nov 02 18:15:00.334646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334566 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.334646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334635 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.334990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334752 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhhk\" (UniqueName: \"kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.334990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334813 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.334990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334887 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtngt\" (UniqueName: \"kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.334990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334910 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb6gm\" (UniqueName: \"kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.334990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334939 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.335206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.334992 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-p4sm2\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.335206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.335039 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjw4q\" (UniqueName: \"kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.335206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.335078 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddkl\" (UniqueName: \"kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl\") pod \"maas-api-key-cleanup-29894055-p4sm2\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.372118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.372075 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2"] Nov 02 18:15:00.436563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436464 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.436563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436536 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-p4sm2\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.436879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436577 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjw4q\" (UniqueName: \"kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.436879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436609 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddkl\" (UniqueName: \"kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl\") pod \"maas-api-key-cleanup-29894055-p4sm2\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.436879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436839 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.437051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436929 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.437051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.436972 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhhk\" (UniqueName: \"kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.437051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437031 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.437230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437115 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtngt\" (UniqueName: \"kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.437230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437142 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb6gm\" (UniqueName: \"kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.437388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437350 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.437463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437423 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-p4sm2\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.437577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437535 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.437577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437574 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.437947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.437931 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.495714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.495612 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddkl\" (UniqueName: \"kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl\") pod \"maas-api-key-cleanup-29894055-p4sm2\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.495714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.495622 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtngt\" (UniqueName: \"kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.496002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.495820 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjw4q\" (UniqueName: \"kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.496126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.496104 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhhk\" (UniqueName: \"kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.496471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.496455 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb6gm\" (UniqueName: \"kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.500330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.500309 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:00.509800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.509775 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:00.541488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.541253 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:00.549414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.548982 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:00.802254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.802139 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:00.952236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.952202 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7"] Nov 02 18:15:00.952938 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:15:00.952912 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55b4a12_e847_4d3a_863d_14540a1e00d4.slice/crio-1b37af001e7c156529cad46502bded8963e5b5ef6ed31ec3a44716d8a0be9836 WatchSource:0}: Error finding container 1b37af001e7c156529cad46502bded8963e5b5ef6ed31ec3a44716d8a0be9836: Status 404 returned error can't find the container with id 1b37af001e7c156529cad46502bded8963e5b5ef6ed31ec3a44716d8a0be9836 Nov 02 18:15:00.955587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.955568 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp"] Nov 02 18:15:00.956641 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:15:00.956618 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcbf67b_2c1b_4dec_92d9_84c2b37c841e.slice/crio-342de35a0e85ebf18fe4f49fc2fc6b3929ebc4dca2cdc2edc214359a1835ead7 WatchSource:0}: Error finding container 342de35a0e85ebf18fe4f49fc2fc6b3929ebc4dca2cdc2edc214359a1835ead7: Status 404 returned error can't find the container with id 342de35a0e85ebf18fe4f49fc2fc6b3929ebc4dca2cdc2edc214359a1835ead7 Nov 02 18:15:00.958534 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:00.958379 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v"] Nov 02 18:15:01.000904 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:15:01.000869 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569d5f2a_8084_4e94_9075_e33abb156c57.slice/crio-2549af4e44af44507af7b179737572c0fdce8c1d0682b9b8b71b69f2e8b72ba9 WatchSource:0}: Error finding container 2549af4e44af44507af7b179737572c0fdce8c1d0682b9b8b71b69f2e8b72ba9: Status 404 returned error can't find the container with id 2549af4e44af44507af7b179737572c0fdce8c1d0682b9b8b71b69f2e8b72ba9 Nov 02 18:15:01.010632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:01.010606 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2"] Nov 02 18:15:01.106101 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:15:01.106069 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542b3673_a988_4875_94cf_b0b099d4080e.slice/crio-94de2f5a66baa97190032051df77d17b9febb762b2e70ceba5bbe5fcc866a1bd WatchSource:0}: Error finding container 94de2f5a66baa97190032051df77d17b9febb762b2e70ceba5bbe5fcc866a1bd: Status 404 returned error can't find the container with id 94de2f5a66baa97190032051df77d17b9febb762b2e70ceba5bbe5fcc866a1bd Nov 02 18:15:01.141330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:01.141294 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" event={"ID":"e55b4a12-e847-4d3a-863d-14540a1e00d4","Type":"ContainerStarted","Data":"1b37af001e7c156529cad46502bded8963e5b5ef6ed31ec3a44716d8a0be9836"} Nov 02 18:15:01.142656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:01.142628 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" event={"ID":"542b3673-a988-4875-94cf-b0b099d4080e","Type":"ContainerStarted","Data":"94de2f5a66baa97190032051df77d17b9febb762b2e70ceba5bbe5fcc866a1bd"} Nov 02 18:15:01.143942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:01.143917 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" event={"ID":"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e","Type":"ContainerStarted","Data":"342de35a0e85ebf18fe4f49fc2fc6b3929ebc4dca2cdc2edc214359a1835ead7"} Nov 02 18:15:01.145084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:01.145063 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" event={"ID":"569d5f2a-8084-4e94-9075-e33abb156c57","Type":"ContainerStarted","Data":"2549af4e44af44507af7b179737572c0fdce8c1d0682b9b8b71b69f2e8b72ba9"} Nov 02 18:15:01.182527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:01.182494 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l"] Nov 02 18:15:01.184669 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:15:01.184640 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21840d99_2971_4c07_927a_fbb9735c378e.slice/crio-0a508c04a2f5635a90086b4907cecd7ba34872dc2e227e40c076187bdbe77825 WatchSource:0}: Error finding container 0a508c04a2f5635a90086b4907cecd7ba34872dc2e227e40c076187bdbe77825: Status 404 returned error can't find the container with id 0a508c04a2f5635a90086b4907cecd7ba34872dc2e227e40c076187bdbe77825 Nov 02 18:15:02.153034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.153000 5647 generic.go:356] "Generic (PLEG): container finished" podID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerID="fc260add4b6695037ab10c802d22368a6ee0427913ee8b7640f8e50411626802" exitCode=60 Nov 02 18:15:02.153225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.153077 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" event={"ID":"e55b4a12-e847-4d3a-863d-14540a1e00d4","Type":"ContainerDied","Data":"fc260add4b6695037ab10c802d22368a6ee0427913ee8b7640f8e50411626802"} Nov 02 18:15:02.153413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.153391 5647 scope.go:122] "RemoveContainer" containerID="fc260add4b6695037ab10c802d22368a6ee0427913ee8b7640f8e50411626802" Nov 02 18:15:02.154783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.154759 5647 generic.go:356] "Generic (PLEG): container finished" podID="21840d99-2971-4c07-927a-fbb9735c378e" containerID="210877cb78ecab33996c4f8412876f1170afaf798d260366ebed482e5ac49484" exitCode=60 Nov 02 18:15:02.154911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.154820 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" event={"ID":"21840d99-2971-4c07-927a-fbb9735c378e","Type":"ContainerDied","Data":"210877cb78ecab33996c4f8412876f1170afaf798d260366ebed482e5ac49484"} Nov 02 18:15:02.154911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.154863 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" event={"ID":"21840d99-2971-4c07-927a-fbb9735c378e","Type":"ContainerStarted","Data":"0a508c04a2f5635a90086b4907cecd7ba34872dc2e227e40c076187bdbe77825"} Nov 02 18:15:02.155069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.155051 5647 scope.go:122] "RemoveContainer" containerID="210877cb78ecab33996c4f8412876f1170afaf798d260366ebed482e5ac49484" Nov 02 18:15:02.156857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.156773 5647 generic.go:356] "Generic (PLEG): container finished" podID="542b3673-a988-4875-94cf-b0b099d4080e" containerID="5c019291a9e8299f08c4013910b96f75c9cf534ca3d525e920eb87c0a3bb00f8" exitCode=60 Nov 02 18:15:02.156939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.156866 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" event={"ID":"542b3673-a988-4875-94cf-b0b099d4080e","Type":"ContainerDied","Data":"5c019291a9e8299f08c4013910b96f75c9cf534ca3d525e920eb87c0a3bb00f8"} Nov 02 18:15:02.157112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.157050 5647 scope.go:122] "RemoveContainer" containerID="5c019291a9e8299f08c4013910b96f75c9cf534ca3d525e920eb87c0a3bb00f8" Nov 02 18:15:02.158642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.158615 5647 generic.go:356] "Generic (PLEG): container finished" podID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerID="6dba8527f4830c8fe912c5de88b546650fe1e5dc101d03147a2ea941940ffa06" exitCode=60 Nov 02 18:15:02.158760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.158693 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" event={"ID":"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e","Type":"ContainerDied","Data":"6dba8527f4830c8fe912c5de88b546650fe1e5dc101d03147a2ea941940ffa06"} Nov 02 18:15:02.158933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.158913 5647 scope.go:122] "RemoveContainer" containerID="6dba8527f4830c8fe912c5de88b546650fe1e5dc101d03147a2ea941940ffa06" Nov 02 18:15:02.160501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.160479 5647 generic.go:356] "Generic (PLEG): container finished" podID="569d5f2a-8084-4e94-9075-e33abb156c57" containerID="041142a9c68bd70ce83f285cea3179799c5b09d6a2a85d4b5eeb419d9f3f2d5a" exitCode=60 Nov 02 18:15:02.160583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.160561 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" event={"ID":"569d5f2a-8084-4e94-9075-e33abb156c57","Type":"ContainerDied","Data":"041142a9c68bd70ce83f285cea3179799c5b09d6a2a85d4b5eeb419d9f3f2d5a"} Nov 02 18:15:02.160725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:02.160713 5647 scope.go:122] "RemoveContainer" containerID="041142a9c68bd70ce83f285cea3179799c5b09d6a2a85d4b5eeb419d9f3f2d5a" Nov 02 18:15:03.168930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.168895 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" event={"ID":"e55b4a12-e847-4d3a-863d-14540a1e00d4","Type":"ContainerStarted","Data":"55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874"} Nov 02 18:15:03.170909 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.170879 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" event={"ID":"21840d99-2971-4c07-927a-fbb9735c378e","Type":"ContainerStarted","Data":"c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0"} Nov 02 18:15:03.173127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.173098 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" event={"ID":"542b3673-a988-4875-94cf-b0b099d4080e","Type":"ContainerStarted","Data":"820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4"} Nov 02 18:15:03.174927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.174896 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" event={"ID":"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e","Type":"ContainerStarted","Data":"3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970"} Nov 02 18:15:03.176642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.176622 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" event={"ID":"569d5f2a-8084-4e94-9075-e33abb156c57","Type":"ContainerStarted","Data":"f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a"} Nov 02 18:15:03.288451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.288394 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" podStartSLOduration=3.288376487 podStartE2EDuration="3.288376487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.2859421 +0000 UTC m=+3090.359469304" watchObservedRunningTime="2026-11-02 18:15:03.288376487 +0000 UTC m=+3090.361903689" Nov 02 18:15:03.402163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.402110 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" podStartSLOduration=3.40209466 podStartE2EDuration="3.40209466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.400384522 +0000 UTC m=+3090.473911726" watchObservedRunningTime="2026-11-02 18:15:03.40209466 +0000 UTC m=+3090.475621891" Nov 02 18:15:03.515414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.515351 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" podStartSLOduration=3.515338784 podStartE2EDuration="3.515338784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.51413133 +0000 UTC m=+3090.587658533" watchObservedRunningTime="2026-11-02 18:15:03.515338784 +0000 UTC m=+3090.588865986" Nov 02 18:15:03.629240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.629184 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" podStartSLOduration=3.629170781 podStartE2EDuration="3.629170781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.627927063 +0000 UTC m=+3090.701454266" watchObservedRunningTime="2026-11-02 18:15:03.629170781 +0000 UTC m=+3090.702697984" Nov 02 18:15:03.744215 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:03.744106 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" podStartSLOduration=3.744091205 podStartE2EDuration="3.744091205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:03.741831023 +0000 UTC m=+3090.815358225" watchObservedRunningTime="2026-11-02 18:15:03.744091205 +0000 UTC m=+3090.817618479" Nov 02 18:15:04.185519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.185483 5647 generic.go:356] "Generic (PLEG): container finished" podID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerID="55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874" exitCode=60 Nov 02 18:15:04.186046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.185556 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" event={"ID":"e55b4a12-e847-4d3a-863d-14540a1e00d4","Type":"ContainerDied","Data":"55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874"} Nov 02 18:15:04.186046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.185634 5647 scope.go:122] "RemoveContainer" containerID="fc260add4b6695037ab10c802d22368a6ee0427913ee8b7640f8e50411626802" Nov 02 18:15:04.186046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.185869 5647 scope.go:122] "RemoveContainer" containerID="55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874" Nov 02 18:15:04.186248 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:04.186215 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7_odh-ai-gateway-infra(e55b4a12-e847-4d3a-863d-14540a1e00d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" Nov 02 18:15:04.187972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.187947 5647 generic.go:356] "Generic (PLEG): container finished" podID="21840d99-2971-4c07-927a-fbb9735c378e" containerID="c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0" exitCode=60 Nov 02 18:15:04.188092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.188042 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" event={"ID":"21840d99-2971-4c07-927a-fbb9735c378e","Type":"ContainerDied","Data":"c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0"} Nov 02 18:15:04.188277 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.188258 5647 scope.go:122] "RemoveContainer" containerID="c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0" Nov 02 18:15:04.188592 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:04.188568 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l_odh-ai-gateway-infra(21840d99-2971-4c07-927a-fbb9735c378e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" podUID="21840d99-2971-4c07-927a-fbb9735c378e" Nov 02 18:15:04.190058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.190039 5647 generic.go:356] "Generic (PLEG): container finished" podID="542b3673-a988-4875-94cf-b0b099d4080e" containerID="820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4" exitCode=60 Nov 02 18:15:04.190164 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.190126 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" event={"ID":"542b3673-a988-4875-94cf-b0b099d4080e","Type":"ContainerDied","Data":"820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4"} Nov 02 18:15:04.190326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.190307 5647 scope.go:122] "RemoveContainer" containerID="820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4" Nov 02 18:15:04.190607 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:04.190581 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-p4sm2_odh-ai-gateway-infra(542b3673-a988-4875-94cf-b0b099d4080e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" podUID="542b3673-a988-4875-94cf-b0b099d4080e" Nov 02 18:15:04.192095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.192071 5647 generic.go:356] "Generic (PLEG): container finished" podID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerID="3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970" exitCode=60 Nov 02 18:15:04.192200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.192146 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" event={"ID":"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e","Type":"ContainerDied","Data":"3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970"} Nov 02 18:15:04.192358 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.192342 5647 scope.go:122] "RemoveContainer" containerID="3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970" Nov 02 18:15:04.192606 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:04.192581 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp_odh-ai-gateway-infra(1dcbf67b-2c1b-4dec-92d9-84c2b37c841e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" Nov 02 18:15:04.194115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.194091 5647 generic.go:356] "Generic (PLEG): container finished" podID="569d5f2a-8084-4e94-9075-e33abb156c57" containerID="f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a" exitCode=60 Nov 02 18:15:04.194202 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.194159 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" event={"ID":"569d5f2a-8084-4e94-9075-e33abb156c57","Type":"ContainerDied","Data":"f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a"} Nov 02 18:15:04.194466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.194453 5647 scope.go:122] "RemoveContainer" containerID="f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a" Nov 02 18:15:04.194717 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:04.194696 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v_odh-ai-gateway-infra(569d5f2a-8084-4e94-9075-e33abb156c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" Nov 02 18:15:04.199694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.199673 5647 scope.go:122] "RemoveContainer" containerID="210877cb78ecab33996c4f8412876f1170afaf798d260366ebed482e5ac49484" Nov 02 18:15:04.214166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.214138 5647 scope.go:122] "RemoveContainer" containerID="5c019291a9e8299f08c4013910b96f75c9cf534ca3d525e920eb87c0a3bb00f8" Nov 02 18:15:04.227674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.227651 5647 scope.go:122] "RemoveContainer" containerID="6dba8527f4830c8fe912c5de88b546650fe1e5dc101d03147a2ea941940ffa06" Nov 02 18:15:04.243370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:04.243347 5647 scope.go:122] "RemoveContainer" containerID="041142a9c68bd70ce83f285cea3179799c5b09d6a2a85d4b5eeb419d9f3f2d5a" Nov 02 18:15:05.202884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:05.202856 5647 scope.go:122] "RemoveContainer" containerID="55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874" Nov 02 18:15:05.203366 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:05.203164 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7_odh-ai-gateway-infra(e55b4a12-e847-4d3a-863d-14540a1e00d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" Nov 02 18:15:05.204477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:05.204448 5647 scope.go:122] "RemoveContainer" containerID="c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0" Nov 02 18:15:05.204781 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:05.204751 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l_odh-ai-gateway-infra(21840d99-2971-4c07-927a-fbb9735c378e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" podUID="21840d99-2971-4c07-927a-fbb9735c378e" Nov 02 18:15:05.206182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:05.206154 5647 scope.go:122] "RemoveContainer" containerID="820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4" Nov 02 18:15:05.206417 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:05.206397 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-p4sm2_odh-ai-gateway-infra(542b3673-a988-4875-94cf-b0b099d4080e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" podUID="542b3673-a988-4875-94cf-b0b099d4080e" Nov 02 18:15:05.207795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:05.207766 5647 scope.go:122] "RemoveContainer" containerID="3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970" Nov 02 18:15:05.208057 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:05.208035 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp_odh-ai-gateway-infra(1dcbf67b-2c1b-4dec-92d9-84c2b37c841e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" Nov 02 18:15:05.209231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:05.209215 5647 scope.go:122] "RemoveContainer" containerID="f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a" Nov 02 18:15:05.209425 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:05.209406 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v_odh-ai-gateway-infra(569d5f2a-8084-4e94-9075-e33abb156c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" Nov 02 18:15:05.950302 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:05.950254 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:15:08.125495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:08.125445 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:15:15.091261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:15.091231 5647 scope.go:122] "RemoveContainer" containerID="3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970" Nov 02 18:15:15.091685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:15.091363 5647 scope.go:122] "RemoveContainer" containerID="c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0" Nov 02 18:15:15.091685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:15.091499 5647 scope.go:122] "RemoveContainer" containerID="55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874" Nov 02 18:15:16.074068 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.074033 5647 scope.go:122] "RemoveContainer" containerID="820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4" Nov 02 18:15:16.284876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.284846 5647 generic.go:356] "Generic (PLEG): container finished" podID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerID="c6c9a2cb33d90026a36fcd40bbce348e8a1698a9c02ba1a957eb7e8a8dc06886" exitCode=60 Nov 02 18:15:16.285329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.284947 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" event={"ID":"e55b4a12-e847-4d3a-863d-14540a1e00d4","Type":"ContainerDied","Data":"c6c9a2cb33d90026a36fcd40bbce348e8a1698a9c02ba1a957eb7e8a8dc06886"} Nov 02 18:15:16.285329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.285010 5647 scope.go:122] "RemoveContainer" containerID="55bd74dabae96eb77d6f76285d7a572635ffa4f9e74758a6051aec787a6cf874" Nov 02 18:15:16.285561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.285538 5647 scope.go:122] "RemoveContainer" containerID="c6c9a2cb33d90026a36fcd40bbce348e8a1698a9c02ba1a957eb7e8a8dc06886" Nov 02 18:15:16.286032 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:16.285952 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7_odh-ai-gateway-infra(e55b4a12-e847-4d3a-863d-14540a1e00d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" Nov 02 18:15:16.288438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.288410 5647 generic.go:356] "Generic (PLEG): container finished" podID="21840d99-2971-4c07-927a-fbb9735c378e" containerID="6c0df87e9bfbb7f898ecd234588a45e90b65ac5a12162a2d9d4cdb12d04c1bf7" exitCode=60 Nov 02 18:15:16.288546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.288445 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" event={"ID":"21840d99-2971-4c07-927a-fbb9735c378e","Type":"ContainerDied","Data":"6c0df87e9bfbb7f898ecd234588a45e90b65ac5a12162a2d9d4cdb12d04c1bf7"} Nov 02 18:15:16.288927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.288902 5647 scope.go:122] "RemoveContainer" containerID="6c0df87e9bfbb7f898ecd234588a45e90b65ac5a12162a2d9d4cdb12d04c1bf7" Nov 02 18:15:16.289239 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:16.289217 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l_odh-ai-gateway-infra(21840d99-2971-4c07-927a-fbb9735c378e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" podUID="21840d99-2971-4c07-927a-fbb9735c378e" Nov 02 18:15:16.291539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.291513 5647 generic.go:356] "Generic (PLEG): container finished" podID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerID="3802bc797046c7ada5e4869b8507e091e694cb056d156c7f530e5bbfd63004cb" exitCode=60 Nov 02 18:15:16.291667 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.291540 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" event={"ID":"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e","Type":"ContainerDied","Data":"3802bc797046c7ada5e4869b8507e091e694cb056d156c7f530e5bbfd63004cb"} Nov 02 18:15:16.291934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.291916 5647 scope.go:122] "RemoveContainer" containerID="3802bc797046c7ada5e4869b8507e091e694cb056d156c7f530e5bbfd63004cb" Nov 02 18:15:16.292202 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:16.292184 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp_odh-ai-gateway-infra(1dcbf67b-2c1b-4dec-92d9-84c2b37c841e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" Nov 02 18:15:16.299095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.299077 5647 scope.go:122] "RemoveContainer" containerID="c6769f96f91ca907b3f9fae76c95cb9ff0fab0d766abdfe3dc6951f626156da0" Nov 02 18:15:16.312315 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:16.312294 5647 scope.go:122] "RemoveContainer" containerID="3eadb765b8ae1d120aaca23280eb6d277be47065ff09cac5d33c8624a245a970" Nov 02 18:15:17.304583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.304499 5647 generic.go:356] "Generic (PLEG): container finished" podID="542b3673-a988-4875-94cf-b0b099d4080e" containerID="801bed1709b588379f848283d4bb371547fa7fc7909d26bc1caf09a056bb7427" exitCode=60 Nov 02 18:15:17.304583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.304563 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" event={"ID":"542b3673-a988-4875-94cf-b0b099d4080e","Type":"ContainerDied","Data":"801bed1709b588379f848283d4bb371547fa7fc7909d26bc1caf09a056bb7427"} Nov 02 18:15:17.305155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.304618 5647 scope.go:122] "RemoveContainer" containerID="820520afeb7a62410cf7d80a881b879e811f0121198d1b07ffc1240bdf7bb8b4" Nov 02 18:15:17.305155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.305061 5647 scope.go:122] "RemoveContainer" containerID="801bed1709b588379f848283d4bb371547fa7fc7909d26bc1caf09a056bb7427" Nov 02 18:15:17.305414 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:17.305388 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-p4sm2_odh-ai-gateway-infra(542b3673-a988-4875-94cf-b0b099d4080e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" podUID="542b3673-a988-4875-94cf-b0b099d4080e" Nov 02 18:15:17.411243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.411206 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7"] Nov 02 18:15:17.635348 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.635311 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l"] Nov 02 18:15:17.654777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.649895 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp"] Nov 02 18:15:17.665947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.665920 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:17.839417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.839384 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm\" (UniqueName: \"kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm\") pod \"e55b4a12-e847-4d3a-863d-14540a1e00d4\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " Nov 02 18:15:17.839610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.839578 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca\") pod \"e55b4a12-e847-4d3a-863d-14540a1e00d4\" (UID: \"e55b4a12-e847-4d3a-863d-14540a1e00d4\") " Nov 02 18:15:17.840959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.840931 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca" pod "e55b4a12-e847-4d3a-863d-14540a1e00d4" (UID: "e55b4a12-e847-4d3a-863d-14540a1e00d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:17.843777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.843706 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm" pod "e55b4a12-e847-4d3a-863d-14540a1e00d4" (UID: "e55b4a12-e847-4d3a-863d-14540a1e00d4"). InnerVolumeSpecName "kube-api-access-rb6gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:17.846137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.846110 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:17.866945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.866922 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:17.966377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.962022 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb6gm\" (UniqueName: \"kubernetes.io/projected/e55b4a12-e847-4d3a-863d-14540a1e00d4-kube-api-access-rb6gm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:17.966377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:17.962060 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55b4a12-e847-4d3a-863d-14540a1e00d4-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.063106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063072 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt\" (UniqueName: \"kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt\") pod \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " Nov 02 18:15:18.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063132 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q\" (UniqueName: \"kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q\") pod \"21840d99-2971-4c07-927a-fbb9735c378e\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " Nov 02 18:15:18.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063180 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca\") pod \"21840d99-2971-4c07-927a-fbb9735c378e\" (UID: \"21840d99-2971-4c07-927a-fbb9735c378e\") " Nov 02 18:15:18.063317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063265 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca\") pod \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\" (UID: \"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e\") " Nov 02 18:15:18.063649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063623 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca" pod "21840d99-2971-4c07-927a-fbb9735c378e" (UID: "21840d99-2971-4c07-927a-fbb9735c378e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:18.063775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063624 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca" pod "1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" (UID: "1dcbf67b-2c1b-4dec-92d9-84c2b37c841e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:18.063892 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063877 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21840d99-2971-4c07-927a-fbb9735c378e-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.063942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.063895 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.065546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.065523 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt" pod "1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" (UID: "1dcbf67b-2c1b-4dec-92d9-84c2b37c841e"). InnerVolumeSpecName "kube-api-access-gtngt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:18.065601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.065556 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q" pod "21840d99-2971-4c07-927a-fbb9735c378e" (UID: "21840d99-2971-4c07-927a-fbb9735c378e"). InnerVolumeSpecName "kube-api-access-fjw4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:18.074347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.074325 5647 scope.go:122] "RemoveContainer" containerID="f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a" Nov 02 18:15:18.165512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.165482 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtngt\" (UniqueName: \"kubernetes.io/projected/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e-kube-api-access-gtngt\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.165512 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.165510 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjw4q\" (UniqueName: \"kubernetes.io/projected/21840d99-2971-4c07-927a-fbb9735c378e-kube-api-access-fjw4q\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.316125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.316044 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" Nov 02 18:15:18.316125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.316055 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7" event={"ID":"e55b4a12-e847-4d3a-863d-14540a1e00d4","Type":"ContainerDied","Data":"1b37af001e7c156529cad46502bded8963e5b5ef6ed31ec3a44716d8a0be9836"} Nov 02 18:15:18.316125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.316122 5647 scope.go:122] "RemoveContainer" containerID="c6c9a2cb33d90026a36fcd40bbce348e8a1698a9c02ba1a957eb7e8a8dc06886" Nov 02 18:15:18.317459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.317427 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" Nov 02 18:15:18.317533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.317462 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l" event={"ID":"21840d99-2971-4c07-927a-fbb9735c378e","Type":"ContainerDied","Data":"0a508c04a2f5635a90086b4907cecd7ba34872dc2e227e40c076187bdbe77825"} Nov 02 18:15:18.321568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.321543 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" event={"ID":"1dcbf67b-2c1b-4dec-92d9-84c2b37c841e","Type":"ContainerDied","Data":"342de35a0e85ebf18fe4f49fc2fc6b3929ebc4dca2cdc2edc214359a1835ead7"} Nov 02 18:15:18.321700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.321631 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp" Nov 02 18:15:18.336070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.336044 5647 scope.go:122] "RemoveContainer" containerID="6c0df87e9bfbb7f898ecd234588a45e90b65ac5a12162a2d9d4cdb12d04c1bf7" Nov 02 18:15:18.347350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.347321 5647 scope.go:122] "RemoveContainer" containerID="3802bc797046c7ada5e4869b8507e091e694cb056d156c7f530e5bbfd63004cb" Nov 02 18:15:18.430687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.430661 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2"] Nov 02 18:15:18.486299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.486267 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l"] Nov 02 18:15:18.495988 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.495953 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-580a21-29894055-c9w2l"] Nov 02 18:15:18.576509 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.576442 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:18.603402 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.603370 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7"] Nov 02 18:15:18.605946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.605922 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894055-9lxn7"] Nov 02 18:15:18.718815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.718778 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp"] Nov 02 18:15:18.721532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.721505 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894055-545xp"] Nov 02 18:15:18.773958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.773931 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca\") pod \"542b3673-a988-4875-94cf-b0b099d4080e\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " Nov 02 18:15:18.774108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.774034 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl\" (UniqueName: \"kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl\") pod \"542b3673-a988-4875-94cf-b0b099d4080e\" (UID: \"542b3673-a988-4875-94cf-b0b099d4080e\") " Nov 02 18:15:18.774333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.774306 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca" pod "542b3673-a988-4875-94cf-b0b099d4080e" (UID: "542b3673-a988-4875-94cf-b0b099d4080e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:18.774480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.774464 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542b3673-a988-4875-94cf-b0b099d4080e-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:18.776216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.776189 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl" pod "542b3673-a988-4875-94cf-b0b099d4080e" (UID: "542b3673-a988-4875-94cf-b0b099d4080e"). InnerVolumeSpecName "kube-api-access-zddkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:18.875220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:18.875185 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddkl\" (UniqueName: \"kubernetes.io/projected/542b3673-a988-4875-94cf-b0b099d4080e-kube-api-access-zddkl\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:19.085621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.085590 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" path="/var/lib/kubelet/pods/1dcbf67b-2c1b-4dec-92d9-84c2b37c841e/volumes" Nov 02 18:15:19.086172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.086144 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21840d99-2971-4c07-927a-fbb9735c378e" path="/var/lib/kubelet/pods/21840d99-2971-4c07-927a-fbb9735c378e/volumes" Nov 02 18:15:19.086566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.086550 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" path="/var/lib/kubelet/pods/e55b4a12-e847-4d3a-863d-14540a1e00d4/volumes" Nov 02 18:15:19.330780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.330745 5647 generic.go:356] "Generic (PLEG): container finished" podID="569d5f2a-8084-4e94-9075-e33abb156c57" containerID="d31bbc4931fc8cce8c61ef2f980b519f64366aae08add762611692099681e040" exitCode=60 Nov 02 18:15:19.331271 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.330807 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" event={"ID":"569d5f2a-8084-4e94-9075-e33abb156c57","Type":"ContainerDied","Data":"d31bbc4931fc8cce8c61ef2f980b519f64366aae08add762611692099681e040"} Nov 02 18:15:19.331271 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.330877 5647 scope.go:122] "RemoveContainer" containerID="f6934b1cfe1246fa5198c2a48d5efebd471959ab3c77b9772097d2e3c6b5188a" Nov 02 18:15:19.331416 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.331300 5647 scope.go:122] "RemoveContainer" containerID="d31bbc4931fc8cce8c61ef2f980b519f64366aae08add762611692099681e040" Nov 02 18:15:19.331681 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:19.331656 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v_odh-ai-gateway-infra(569d5f2a-8084-4e94-9075-e33abb156c57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" Nov 02 18:15:19.334328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.334309 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" event={"ID":"542b3673-a988-4875-94cf-b0b099d4080e","Type":"ContainerDied","Data":"94de2f5a66baa97190032051df77d17b9febb762b2e70ceba5bbe5fcc866a1bd"} Nov 02 18:15:19.334328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.334321 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2" Nov 02 18:15:19.343960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.343941 5647 scope.go:122] "RemoveContainer" containerID="801bed1709b588379f848283d4bb371547fa7fc7909d26bc1caf09a056bb7427" Nov 02 18:15:19.561192 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.561095 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2"] Nov 02 18:15:19.568603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:19.568576 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-p4sm2"] Nov 02 18:15:20.457435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.457401 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v"] Nov 02 18:15:20.634112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.634084 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:20.706099 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.706065 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk\" (UniqueName: \"kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk\") pod \"569d5f2a-8084-4e94-9075-e33abb156c57\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " Nov 02 18:15:20.706340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.706322 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca\") pod \"569d5f2a-8084-4e94-9075-e33abb156c57\" (UID: \"569d5f2a-8084-4e94-9075-e33abb156c57\") " Nov 02 18:15:20.706657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.706628 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca" pod "569d5f2a-8084-4e94-9075-e33abb156c57" (UID: "569d5f2a-8084-4e94-9075-e33abb156c57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:20.706849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.706832 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d5f2a-8084-4e94-9075-e33abb156c57-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:20.708585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.708532 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk" pod "569d5f2a-8084-4e94-9075-e33abb156c57" (UID: "569d5f2a-8084-4e94-9075-e33abb156c57"). InnerVolumeSpecName "kube-api-access-9hhhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:20.808429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:20.808397 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhhk\" (UniqueName: \"kubernetes.io/projected/569d5f2a-8084-4e94-9075-e33abb156c57-kube-api-access-9hhhk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:21.082636 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:21.082554 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542b3673-a988-4875-94cf-b0b099d4080e" path="/var/lib/kubelet/pods/542b3673-a988-4875-94cf-b0b099d4080e/volumes" Nov 02 18:15:21.352296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:21.352263 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" Nov 02 18:15:21.352491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:21.352292 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v" event={"ID":"569d5f2a-8084-4e94-9075-e33abb156c57","Type":"ContainerDied","Data":"2549af4e44af44507af7b179737572c0fdce8c1d0682b9b8b71b69f2e8b72ba9"} Nov 02 18:15:21.352491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:21.352358 5647 scope.go:122] "RemoveContainer" containerID="d31bbc4931fc8cce8c61ef2f980b519f64366aae08add762611692099681e040" Nov 02 18:15:21.521884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:21.521849 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v"] Nov 02 18:15:21.572393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:21.572361 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c82472-29894055-5bg8v"] Nov 02 18:15:23.082230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:23.082200 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" path="/var/lib/kubelet/pods/569d5f2a-8084-4e94-9075-e33abb156c57/volumes" Nov 02 18:15:27.611338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:27.611302 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j"] Nov 02 18:15:27.611764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:27.611523 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" containerID="cri-o://7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886" gracePeriod=30 Nov 02 18:15:32.877840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.877768 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:15:32.972494 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972455 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972544 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972613 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972645 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972691 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972834 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972881 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:32.972975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972931 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.972975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.972938 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:32.973197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973036 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:32.973197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973070 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.973197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973141 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token\") pod \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\" (UID: \"9f8ec92c-b980-4ac6-b145-123e091d1e5d\") " Nov 02 18:15:32.973197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973134 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:32.973377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973299 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:32.973914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973887 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:32.974051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973918 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:32.974051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973935 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:32.974051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973948 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:32.974051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.973961 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:32.975103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.975069 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:32.975362 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.975342 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:32.975442 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.975426 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:32.975499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:32.975462 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v" pod "9f8ec92c-b980-4ac6-b145-123e091d1e5d" (UID: "9f8ec92c-b980-4ac6-b145-123e091d1e5d"). InnerVolumeSpecName "kube-api-access-m689v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:33.074759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.074706 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:33.074938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.074769 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:33.074938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.074782 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f8ec92c-b980-4ac6-b145-123e091d1e5d-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:33.074938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.074792 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m689v\" (UniqueName: \"kubernetes.io/projected/9f8ec92c-b980-4ac6-b145-123e091d1e5d-kube-api-access-m689v\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:33.445625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.445586 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerID="7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886" exitCode=0 Nov 02 18:15:33.445861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.445658 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" event={"ID":"9f8ec92c-b980-4ac6-b145-123e091d1e5d","Type":"ContainerDied","Data":"7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886"} Nov 02 18:15:33.445861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.445693 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" event={"ID":"9f8ec92c-b980-4ac6-b145-123e091d1e5d","Type":"ContainerDied","Data":"d41e1cd54938355378c961125d9eea55b4493d73c2aa86af838b0bccdb512eb8"} Nov 02 18:15:33.445861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.445714 5647 scope.go:122] "RemoveContainer" containerID="7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886" Nov 02 18:15:33.446066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.445896 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" Nov 02 18:15:33.458524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.458500 5647 scope.go:122] "RemoveContainer" containerID="7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886" Nov 02 18:15:33.458868 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:33.458841 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886\": container with ID starting with 7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886 not found: ID does not exist" containerID="7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886" Nov 02 18:15:33.458969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.458880 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886"} err="failed to get container status \"7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886\": rpc error: code = NotFound desc = could not find container \"7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886\": container with ID starting with 7baac2bffbd5b9ea8f35a46d654ed8fc86f16c787151e59ef9aa6626f9319886 not found: ID does not exist" Nov 02 18:15:33.621962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.621902 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j"] Nov 02 18:15:33.624366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.624343 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j"] Nov 02 18:15:33.812328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.812242 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:15:33.848243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.848208 5647 patch_prober.go:29] interesting pod/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:15:33.848454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:33.848285 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-jx85j" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:15:35.082294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:35.082257 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" path="/var/lib/kubelet/pods/9f8ec92c-b980-4ac6-b145-123e091d1e5d/volumes" Nov 02 18:15:35.952965 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:15:35.952924 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:15:37.294327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:37.294247 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg"] Nov 02 18:15:37.294763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:37.294523 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" containerID="cri-o://cd08f2bd63af1bf8434fa22552d02c76da42891e943852100fc5428c328d462b" gracePeriod=30 Nov 02 18:15:42.510002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.509949 5647 generic.go:356] "Generic (PLEG): container finished" podID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerID="cd08f2bd63af1bf8434fa22552d02c76da42891e943852100fc5428c328d462b" exitCode=0 Nov 02 18:15:42.510484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.510031 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" event={"ID":"080bf38f-c034-4ccd-8c09-e004c25578e5","Type":"ContainerDied","Data":"cd08f2bd63af1bf8434fa22552d02c76da42891e943852100fc5428c328d462b"} Nov 02 18:15:42.575584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.575491 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:15:42.625937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.625904 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.625965 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.625999 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626058 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626097 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626122 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626160 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626241 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:42.626444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626325 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626372 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:42.626661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626461 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token\") pod \"080bf38f-c034-4ccd-8c09-e004c25578e5\" (UID: \"080bf38f-c034-4ccd-8c09-e004c25578e5\") " Nov 02 18:15:42.626775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626462 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:42.626775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626510 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:15:42.626889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.626722 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:42.627117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.627087 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.627246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.627122 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.627246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.627177 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/080bf38f-c034-4ccd-8c09-e004c25578e5-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.627367 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.627294 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.627367 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.627324 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.628948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.628904 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "kube-api-access-blfrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:42.629059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.628961 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:15:42.629141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.629111 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:15:42.629251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.629197 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token" pod "080bf38f-c034-4ccd-8c09-e004c25578e5" (UID: "080bf38f-c034-4ccd-8c09-e004c25578e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:15:42.728435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.728402 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.728435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.728432 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.728435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.728443 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfrb\" (UniqueName: \"kubernetes.io/projected/080bf38f-c034-4ccd-8c09-e004c25578e5-kube-api-access-blfrb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:42.728435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:42.728452 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/080bf38f-c034-4ccd-8c09-e004c25578e5-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:15:43.524920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:43.524810 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" event={"ID":"080bf38f-c034-4ccd-8c09-e004c25578e5","Type":"ContainerDied","Data":"31adc099e22604da3a9b865415ed54c97011d66f8b92d150db64760004141a10"} Nov 02 18:15:43.525388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:43.524940 5647 scope.go:122] "RemoveContainer" containerID="cd08f2bd63af1bf8434fa22552d02c76da42891e943852100fc5428c328d462b" Nov 02 18:15:43.525388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:43.525161 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg" Nov 02 18:15:43.692675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:43.692637 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg"] Nov 02 18:15:43.744064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:43.744029 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-p64kg"] Nov 02 18:15:44.532966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:44.532929 5647 generic.go:356] "Generic (PLEG): container finished" podID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerID="8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029" exitCode=0 Nov 02 18:15:44.533471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:44.533006 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" event={"ID":"289c4ea7-7005-4b14-92c3-f963d3a7242e","Type":"ContainerDied","Data":"8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029"} Nov 02 18:15:44.533543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:44.533503 5647 scope.go:122] "RemoveContainer" containerID="8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029" Nov 02 18:15:44.535314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:44.535286 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:15:44.535396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:44.535357 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:15:44.535396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:44.535390 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:15:45.087320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:45.087291 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" path="/var/lib/kubelet/pods/080bf38f-c034-4ccd-8c09-e004c25578e5/volumes" Nov 02 18:15:45.542456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:45.542370 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" event={"ID":"289c4ea7-7005-4b14-92c3-f963d3a7242e","Type":"ContainerStarted","Data":"6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f"} Nov 02 18:15:45.722183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:45.722143 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:15:45.722550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:45.722528 5647 patch_prober.go:29] interesting pod/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 02 18:15:45.723010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:45.722709 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 02 18:15:46.726920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:46.726879 5647 patch_prober.go:29] interesting pod/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 02 18:15:46.727365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:46.726950 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 02 18:15:47.723211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:47.723166 5647 patch_prober.go:29] interesting pod/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 02 18:15:47.723515 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:47.723484 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 02 18:15:48.724758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:48.724158 5647 patch_prober.go:29] interesting pod/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 02 18:15:48.724758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:48.724219 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 02 18:15:49.721193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:49.721153 5647 patch_prober.go:29] interesting pod/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 02 18:15:49.721975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:49.721943 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 02 18:15:50.721758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:50.720846 5647 patch_prober.go:29] interesting pod/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Nov 02 18:15:50.721758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:50.720940 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Nov 02 18:15:51.723052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:51.723012 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:15:51.723561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:51.723395 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:15:51.724216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:15:51.724193 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:16:02.080199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080160 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m"] Nov 02 18:16:02.080752 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080717 5647 cpu_manager.go:414] "containerMap: removing container" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080747 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080775 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080780 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080792 5647 cpu_manager.go:414] "containerMap: removing container" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080798 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080805 5647 cpu_manager.go:414] "containerMap: removing container" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080810 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:16:02.080807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080819 5647 cpu_manager.go:414] "containerMap: removing container" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080824 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080840 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080845 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080853 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080857 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080862 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080867 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080877 5647 cpu_manager.go:414] "containerMap: removing container" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080882 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080893 5647 cpu_manager.go:414] "containerMap: removing container" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080899 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080905 5647 cpu_manager.go:414] "containerMap: removing container" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080909 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080934 5647 cpu_manager.go:414] "containerMap: removing container" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080938 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080945 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.080950 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081045 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081052 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f8ec92c-b980-4ac6-b145-123e091d1e5d" containerName="istio-proxy" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081061 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081066 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081071 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081078 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081083 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="21840d99-2971-4c07-927a-fbb9735c378e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081090 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081097 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:16:02.081092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081104 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="080bf38f-c034-4ccd-8c09-e004c25578e5" containerName="istio-proxy" Nov 02 18:16:02.082030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081111 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.082030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081116 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:16:02.082030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081122 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:16:02.082030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.081130 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:16:02.084395 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.084378 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.189520 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.189489 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-eb709f-serving-cert\"" Nov 02 18:16:02.193763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.193716 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m"] Nov 02 18:16:02.200270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.200232 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.200460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.200294 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dp5w\" (UniqueName: \"kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.200460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.200423 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.200579 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.200500 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.301227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.301194 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.301435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.301321 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.301435 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:16:02.301340 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-serving-cert: secret "maas-api-e2e-autores-eb709f-serving-cert" not found Nov 02 18:16:02.301435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.301346 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dp5w\" (UniqueName: \"kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.301435 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:16:02.301406 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls podName:9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9 nodeName:}" failed. No retries permitted until 2026-11-02 18:16:02.801383057 +0000 UTC m=+3149.874910242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls") pod "maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" (UID: "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9") : secret "maas-api-e2e-autores-eb709f-serving-cert" not found Nov 02 18:16:02.301705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.301492 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.302349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.302325 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.303886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.303863 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.365521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.365481 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dp5w\" (UniqueName: \"kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.806925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.806836 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.809422 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.809404 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\") pod \"maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:02.997690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:02.997654 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:03.377283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:03.377247 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m"] Nov 02 18:16:03.377911 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:16:03.377878 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5d8ca4_b88f_4bd6_a767_3d8c045dfef9.slice/crio-d36c3e24e383c69e31527e14c84767ce13f160692335f88d48231b5b96cb6cc2 WatchSource:0}: Error finding container d36c3e24e383c69e31527e14c84767ce13f160692335f88d48231b5b96cb6cc2: Status 404 returned error can't find the container with id d36c3e24e383c69e31527e14c84767ce13f160692335f88d48231b5b96cb6cc2 Nov 02 18:16:03.388527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:03.388492 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:16:03.658672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:03.658573 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" event={"ID":"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9","Type":"ContainerStarted","Data":"d36c3e24e383c69e31527e14c84767ce13f160692335f88d48231b5b96cb6cc2"} Nov 02 18:16:04.666313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:04.666271 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" event={"ID":"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9","Type":"ContainerStarted","Data":"c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b"} Nov 02 18:16:04.666727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:04.666353 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:04.785232 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:04.785168 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" podStartSLOduration=1.9856030009999999 podStartE2EDuration="2.785153004s" totalImagesPullingTime="799.550003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:04.783225448 +0000 UTC m=+3151.856752651" watchObservedRunningTime="2026-11-02 18:16:04.785153004 +0000 UTC m=+3151.858680203" Nov 02 18:16:05.949497 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:16:05.949452 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:16:08.193482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:08.193448 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:16:10.677654 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:10.677619 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:16:35.956258 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:16:35.956164 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:16:40.906146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:40.906112 5647 generic.go:356] "Generic (PLEG): container finished" podID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerID="1a43bdfc771d0d5d1893f35c7f949a08f68255825657e1208a21e130de09f606" exitCode=0 Nov 02 18:16:40.906607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:40.906151 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" event={"ID":"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05","Type":"ContainerDied","Data":"1a43bdfc771d0d5d1893f35c7f949a08f68255825657e1208a21e130de09f606"} Nov 02 18:16:40.906607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:40.906541 5647 scope.go:122] "RemoveContainer" containerID="1a43bdfc771d0d5d1893f35c7f949a08f68255825657e1208a21e130de09f606" Nov 02 18:16:40.908187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:40.908151 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:16:40.908320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:40.908241 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:16:40.908320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:40.908293 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:16:41.934006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:41.933964 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" event={"ID":"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05","Type":"ContainerStarted","Data":"053cd1823c7f8b826be377b8ccc7f8470a714443dc4968e4b79f0fbfbc30d999"} Nov 02 18:16:42.654586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:42.654549 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:16:42.654951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:42.654924 5647 patch_prober.go:29] interesting pod/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 02 18:16:42.655066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:42.654987 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 02 18:16:43.655321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:43.655280 5647 patch_prober.go:29] interesting pod/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 02 18:16:43.655831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:43.655337 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 02 18:16:44.655529 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:44.655490 5647 patch_prober.go:29] interesting pod/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 02 18:16:44.656076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:44.655554 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 02 18:16:45.654504 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:45.654459 5647 patch_prober.go:29] interesting pod/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 02 18:16:45.654700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:45.654537 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 02 18:16:46.654796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:46.654751 5647 patch_prober.go:29] interesting pod/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Nov 02 18:16:46.655310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:46.654846 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Nov 02 18:16:47.660012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:47.659981 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:16:47.660510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:47.660492 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:16:47.661347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:47.661326 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:16:54.971550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:16:54.971519 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:17:05.948904 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:17:05.948861 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:17:35.949226 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:17:35.949186 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:17:55.195410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:17:55.195328 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:18:00.800029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.799985 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg"] Nov 02 18:18:00.800683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800667 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:18:00.800748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800685 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dcbf67b-2c1b-4dec-92d9-84c2b37c841e" containerName="cleanup" Nov 02 18:18:00.800748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800705 5647 cpu_manager.go:414] "containerMap: removing container" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:18:00.800748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800710 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:18:00.800748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800719 5647 cpu_manager.go:414] "containerMap: removing container" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:18:00.800748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800724 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:18:00.800924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800767 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:18:00.800924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800773 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:18:00.800924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800892 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="569d5f2a-8084-4e94-9075-e33abb156c57" containerName="cleanup" Nov 02 18:18:00.800924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800904 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="542b3673-a988-4875-94cf-b0b099d4080e" containerName="cleanup" Nov 02 18:18:00.800924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.800916 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55b4a12-e847-4d3a-863d-14540a1e00d4" containerName="cleanup" Nov 02 18:18:00.804555 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.804531 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.857874 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.857836 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a8da6e-openshift-default-dockercfg-2d7bx\"" Nov 02 18:18:00.911831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.911791 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg"] Nov 02 18:18:00.921002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.920956 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921132 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921192 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921226 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921306 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921428 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921520 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921473 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921520 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921496 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztzs7\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:00.921627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:00.921613 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022358 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022429 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022451 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022477 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022569 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022595 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022620 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztzs7\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022713 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022804 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.022999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.022965 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.023183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.023053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.023183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.023129 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.023304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.023253 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.023304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.023255 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.025125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.025098 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.025441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.025421 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.080892 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.080816 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.081145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.081122 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztzs7\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7\") pod \"e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.118445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.118407 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:01.576269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.576239 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg"] Nov 02 18:18:01.577266 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:18:01.577242 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9166951_3f12_47ff_b3a9_75c1d616f309.slice/crio-586207df4d7c4f2a4ce08752c0fd6d4c22067e860aee0ecc23119a09456e3cef WatchSource:0}: Error finding container 586207df4d7c4f2a4ce08752c0fd6d4c22067e860aee0ecc23119a09456e3cef: Status 404 returned error can't find the container with id 586207df4d7c4f2a4ce08752c0fd6d4c22067e860aee0ecc23119a09456e3cef Nov 02 18:18:01.579831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.579800 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:18:01.579908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.579864 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:18:01.579908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:01.579892 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:18:02.474299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:02.474260 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" event={"ID":"f9166951-3f12-47ff-b3a9-75c1d616f309","Type":"ContainerStarted","Data":"244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329"} Nov 02 18:18:02.474716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:02.474323 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" event={"ID":"f9166951-3f12-47ff-b3a9-75c1d616f309","Type":"ContainerStarted","Data":"586207df4d7c4f2a4ce08752c0fd6d4c22067e860aee0ecc23119a09456e3cef"} Nov 02 18:18:02.661774 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:02.661695 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podStartSLOduration=2.661675728 podStartE2EDuration="2.661675728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:02.659500696 +0000 UTC m=+3269.733027901" watchObservedRunningTime="2026-11-02 18:18:02.661675728 +0000 UTC m=+3269.735202934" Nov 02 18:18:03.119284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:03.119245 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:03.125155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:03.125129 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:03.411054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:03.410972 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72"] Nov 02 18:18:03.480887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:03.480851 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:03.481955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:03.481937 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:18:03.513812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:03.513777 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt"] Nov 02 18:18:05.949257 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:18:05.949202 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:18:08.411453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.411408 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" podUID="72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" containerName="payload-processing" containerID="cri-o://98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d" gracePeriod=25 Nov 02 18:18:08.514677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.514638 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" podUID="47c319ea-ef07-460d-80a1-5713420b9d6f" containerName="payload-pre-processing" containerID="cri-o://44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93" gracePeriod=25 Nov 02 18:18:08.698812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.698783 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:18:08.775189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.775161 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:18:08.839643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.839602 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9\" (UniqueName: \"kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9\") pod \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " Nov 02 18:18:08.839916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.839883 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume\") pod \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\" (UID: \"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0\") " Nov 02 18:18:08.840553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.840526 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume" pod "72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" (UID: "72b435d0-c9a9-4bd9-97c5-dbc4e40379f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:18:08.842561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.842538 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9" pod "72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" (UID: "72b435d0-c9a9-4bd9-97c5-dbc4e40379f0"). InnerVolumeSpecName "kube-api-access-sg4n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:18:08.941410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.941331 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4\" (UniqueName: \"kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4\") pod \"47c319ea-ef07-460d-80a1-5713420b9d6f\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " Nov 02 18:18:08.941410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.941392 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume\") pod \"47c319ea-ef07-460d-80a1-5713420b9d6f\" (UID: \"47c319ea-ef07-460d-80a1-5713420b9d6f\") " Nov 02 18:18:08.941884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.941859 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume" pod "47c319ea-ef07-460d-80a1-5713420b9d6f" (UID: "47c319ea-ef07-460d-80a1-5713420b9d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:18:08.942035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.942020 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:08.942077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.942038 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47c319ea-ef07-460d-80a1-5713420b9d6f-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:08.942077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.942047 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4n9\" (UniqueName: \"kubernetes.io/projected/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0-kube-api-access-sg4n9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:08.943832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:08.943801 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4" pod "47c319ea-ef07-460d-80a1-5713420b9d6f" (UID: "47c319ea-ef07-460d-80a1-5713420b9d6f"). InnerVolumeSpecName "kube-api-access-6pjv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:18:09.043978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.043931 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjv4\" (UniqueName: \"kubernetes.io/projected/47c319ea-ef07-460d-80a1-5713420b9d6f-kube-api-access-6pjv4\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:09.524769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.524677 5647 generic.go:356] "Generic (PLEG): container finished" podID="72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" containerID="98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d" exitCode=0 Nov 02 18:18:09.525315 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.524927 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" Nov 02 18:18:09.525315 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.525249 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" event={"ID":"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0","Type":"ContainerDied","Data":"98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d"} Nov 02 18:18:09.525449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.525373 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72" event={"ID":"72b435d0-c9a9-4bd9-97c5-dbc4e40379f0","Type":"ContainerDied","Data":"7f38baf35d40a37601e5f854d38e0192879a2c8a2ec81f4364bccdb3678e54c9"} Nov 02 18:18:09.525449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.525411 5647 scope.go:122] "RemoveContainer" containerID="98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d" Nov 02 18:18:09.527700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.527663 5647 generic.go:356] "Generic (PLEG): container finished" podID="47c319ea-ef07-460d-80a1-5713420b9d6f" containerID="44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93" exitCode=0 Nov 02 18:18:09.528049 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.527884 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" event={"ID":"47c319ea-ef07-460d-80a1-5713420b9d6f","Type":"ContainerDied","Data":"44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93"} Nov 02 18:18:09.528145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.528089 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" event={"ID":"47c319ea-ef07-460d-80a1-5713420b9d6f","Type":"ContainerDied","Data":"986f78baa3b00723b280396e956e2ce87bd5bdc60829e4c2fd2989f98a966678"} Nov 02 18:18:09.528258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.528241 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt" Nov 02 18:18:09.541773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.541726 5647 scope.go:122] "RemoveContainer" containerID="98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d" Nov 02 18:18:09.542177 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:18:09.542149 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d\": container with ID starting with 98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d not found: ID does not exist" containerID="98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d" Nov 02 18:18:09.542234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.542193 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d"} err="failed to get container status \"98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d\": rpc error: code = NotFound desc = could not find container \"98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d\": container with ID starting with 98f5ab36532bfa92bc59c98a7bd9d9450b8ea1c5d679eb60c1097be03102dc0d not found: ID does not exist" Nov 02 18:18:09.542234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.542218 5647 scope.go:122] "RemoveContainer" containerID="44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93" Nov 02 18:18:09.555869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.555845 5647 scope.go:122] "RemoveContainer" containerID="44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93" Nov 02 18:18:09.556261 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:18:09.556241 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93\": container with ID starting with 44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93 not found: ID does not exist" containerID="44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93" Nov 02 18:18:09.556330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.556271 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93"} err="failed to get container status \"44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93\": rpc error: code = NotFound desc = could not find container \"44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93\": container with ID starting with 44f0b09e1fe69eec97597bb8f325a1353c7fc2fb1609cd25b45219f27a27dc93 not found: ID does not exist" Nov 02 18:18:09.694055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.693954 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt"] Nov 02 18:18:09.697057 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.696996 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3dc055-647fdb46f5-mk2xt"] Nov 02 18:18:09.807629 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.807598 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72"] Nov 02 18:18:09.812462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:09.812430 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-3dc055-6b94f7ff85-6kn72"] Nov 02 18:18:11.082538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:11.082502 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47c319ea-ef07-460d-80a1-5713420b9d6f" path="/var/lib/kubelet/pods/47c319ea-ef07-460d-80a1-5713420b9d6f/volumes" Nov 02 18:18:11.082973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:11.082927 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" path="/var/lib/kubelet/pods/72b435d0-c9a9-4bd9-97c5-dbc4e40379f0/volumes" Nov 02 18:18:22.401077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:22.401025 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg"] Nov 02 18:18:34.413189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:34.413158 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:18:34.418365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:34.418342 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:18:35.949996 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:18:35.949948 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:18:37.423336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.423265 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" podUID="ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" containerName="main" containerID="cri-o://bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7" gracePeriod=45 Nov 02 18:18:37.704367 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.704340 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:18:37.730804 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.730766 5647 generic.go:356] "Generic (PLEG): container finished" podID="ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" containerID="bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7" exitCode=0 Nov 02 18:18:37.730804 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.730801 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" Nov 02 18:18:37.731040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.730837 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" event={"ID":"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9","Type":"ContainerDied","Data":"bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7"} Nov 02 18:18:37.731040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.730878 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg" event={"ID":"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9","Type":"ContainerDied","Data":"1bd4fcdfdf148bc20a1dbe66e540ddc86444419b77609c7973ba8e8e10807641"} Nov 02 18:18:37.731040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.730898 5647 scope.go:122] "RemoveContainer" containerID="bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7" Nov 02 18:18:37.740574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.740553 5647 scope.go:122] "RemoveContainer" containerID="bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7" Nov 02 18:18:37.740889 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:18:37.740869 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7\": container with ID starting with bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7 not found: ID does not exist" containerID="bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7" Nov 02 18:18:37.740954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.740901 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7"} err="failed to get container status \"bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7\": rpc error: code = NotFound desc = could not find container \"bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7\": container with ID starting with bf8abe0052007c77b837d37e44b6f48706daaf824973449b5b6e563a938d72a7 not found: ID does not exist" Nov 02 18:18:37.834070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.834039 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6\" (UniqueName: \"kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6\") pod \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " Nov 02 18:18:37.834698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.834141 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm\") pod \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " Nov 02 18:18:37.834698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.834371 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs\") pod \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " Nov 02 18:18:37.834698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.834425 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache\") pod \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " Nov 02 18:18:37.834698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.834467 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir\") pod \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " Nov 02 18:18:37.834698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.834496 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home\") pod \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\" (UID: \"ce11cc88-9b1c-4708-822e-d9ecd6ce29c9\") " Nov 02 18:18:37.835165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.835021 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache" pod "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" (UID: "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:18:37.835274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.835250 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir" pod "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" (UID: "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:18:37.835336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.835316 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home" pod "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" (UID: "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:18:37.836769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.836722 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6" pod "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" (UID: "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9"). InnerVolumeSpecName "kube-api-access-n4ql6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:18:37.837385 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.837366 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm" pod "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" (UID: "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:18:37.837467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.837417 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs" pod "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" (UID: "ce11cc88-9b1c-4708-822e-d9ecd6ce29c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:18:37.936333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.936247 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:37.936333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.936285 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:37.936333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.936302 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:37.936333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.936312 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4ql6\" (UniqueName: \"kubernetes.io/projected/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-kube-api-access-n4ql6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:37.936333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.936323 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:37.936333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:37.936332 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:18:38.152654 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:38.152621 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg"] Nov 02 18:18:38.155308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:38.155287 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-0ad26a/auth-test-model-0ad26a-kserve-6f79db4d9c-m5jlg"] Nov 02 18:18:39.082724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:39.082689 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" path="/var/lib/kubelet/pods/ce11cc88-9b1c-4708-822e-d9ecd6ce29c9/volumes" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.181014 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh"] Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.181912 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" containerName="main" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.181932 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" containerName="main" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.181966 5647 cpu_manager.go:414] "containerMap: removing container" podUID="72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" containerName="payload-processing" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.181973 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" containerName="payload-processing" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.181999 5647 cpu_manager.go:414] "containerMap: removing container" podUID="47c319ea-ef07-460d-80a1-5713420b9d6f" containerName="payload-pre-processing" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.182006 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47c319ea-ef07-460d-80a1-5713420b9d6f" containerName="payload-pre-processing" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.182145 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b435d0-c9a9-4bd9-97c5-dbc4e40379f0" containerName="payload-processing" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.182160 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="47c319ea-ef07-460d-80a1-5713420b9d6f" containerName="payload-pre-processing" Nov 02 18:18:42.183404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.182177 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce11cc88-9b1c-4708-822e-d9ecd6ce29c9" containerName="main" Nov 02 18:18:42.187749 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.187705 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.240235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.240200 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh"] Nov 02 18:18:42.288359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.288326 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cd738b-dockercfg-465k9\"" Nov 02 18:18:42.288538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.288426 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cd738b\"" Nov 02 18:18:42.305619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.305590 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72vz\" (UniqueName: \"kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz\") pod \"payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.305831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.305808 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.407108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.407069 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.407297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.407188 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72vz\" (UniqueName: \"kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz\") pod \"payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.407902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.407876 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.466146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.466082 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72vz\" (UniqueName: \"kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz\") pod \"payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.519591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.519558 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:42.897933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:42.897908 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh"] Nov 02 18:18:42.899081 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:18:42.899053 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dc1ad5_e093_4a8b_830b_87632bd8b912.slice/crio-e9a2c45173e1eb197f51fe3425c0b95330d8c76aa165637a69491600b104feec WatchSource:0}: Error finding container e9a2c45173e1eb197f51fe3425c0b95330d8c76aa165637a69491600b104feec: Status 404 returned error can't find the container with id e9a2c45173e1eb197f51fe3425c0b95330d8c76aa165637a69491600b104feec Nov 02 18:18:43.811928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:43.811880 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" event={"ID":"72dc1ad5-e093-4a8b-830b-87632bd8b912","Type":"ContainerStarted","Data":"d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976"} Nov 02 18:18:43.812419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:43.811949 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:18:43.812419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:43.811968 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" event={"ID":"72dc1ad5-e093-4a8b-830b-87632bd8b912","Type":"ContainerStarted","Data":"e9a2c45173e1eb197f51fe3425c0b95330d8c76aa165637a69491600b104feec"} Nov 02 18:18:43.932286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:43.932228 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" podStartSLOduration=1.932214551 podStartE2EDuration="1.932214551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:43.930524278 +0000 UTC m=+3311.004051484" watchObservedRunningTime="2026-11-02 18:18:43.932214551 +0000 UTC m=+3311.005741753" Nov 02 18:18:54.820921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:18:54.820891 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:19:05.961412 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:19:05.961371 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:19:07.244938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:07.244900 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d"] Nov 02 18:19:07.245374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:07.245172 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" containerID="cri-o://6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f" gracePeriod=30 Nov 02 18:19:12.516110 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.516035 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:19:12.527056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.527031 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584175 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584303 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584340 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584378 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584418 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584480 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584512 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584558 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.585761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.584748 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj\") pod \"289c4ea7-7005-4b14-92c3-f963d3a7242e\" (UID: \"289c4ea7-7005-4b14-92c3-f963d3a7242e\") " Nov 02 18:19:12.590273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.590006 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:19:12.590758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.590531 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:19:12.591360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.591243 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:19:12.595240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.594872 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:19:12.607753 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.604048 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:19:12.608489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.608424 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:19:12.654199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.654162 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "kube-api-access-kzmdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:19:12.656283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.656240 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:19:12.658262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.658214 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo" pod "289c4ea7-7005-4b14-92c3-f963d3a7242e" (UID: "289c4ea7-7005-4b14-92c3-f963d3a7242e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.691999 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692035 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/289c4ea7-7005-4b14-92c3-f963d3a7242e-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692051 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692065 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmdj\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-kube-api-access-kzmdj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692080 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692093 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692108 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/289c4ea7-7005-4b14-92c3-f963d3a7242e-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692123 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:12.694050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:12.692137 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/289c4ea7-7005-4b14-92c3-f963d3a7242e-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:19:13.003602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.003565 5647 generic.go:356] "Generic (PLEG): container finished" podID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerID="6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f" exitCode=0 Nov 02 18:19:13.003947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.003625 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" Nov 02 18:19:13.003947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.003783 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" event={"ID":"289c4ea7-7005-4b14-92c3-f963d3a7242e","Type":"ContainerDied","Data":"6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f"} Nov 02 18:19:13.003947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.003835 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d" event={"ID":"289c4ea7-7005-4b14-92c3-f963d3a7242e","Type":"ContainerDied","Data":"dac81dbce77b1c38c3d7a0d933e601680da7c699133e446ae7dca18b5041089c"} Nov 02 18:19:13.003947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.003863 5647 scope.go:122] "RemoveContainer" containerID="6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f" Nov 02 18:19:13.018240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.018214 5647 scope.go:122] "RemoveContainer" containerID="8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029" Nov 02 18:19:13.028939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.028919 5647 scope.go:122] "RemoveContainer" containerID="6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f" Nov 02 18:19:13.029283 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:19:13.029262 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f\": container with ID starting with 6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f not found: ID does not exist" containerID="6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f" Nov 02 18:19:13.029351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.029293 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f"} err="failed to get container status \"6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f\": rpc error: code = NotFound desc = could not find container \"6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f\": container with ID starting with 6294e0f4e6e31a405690c72b993e645e96b0b633f042dde24b8136d175b9f38f not found: ID does not exist" Nov 02 18:19:13.029351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.029311 5647 scope.go:122] "RemoveContainer" containerID="8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029" Nov 02 18:19:13.029620 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:19:13.029598 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029\": container with ID starting with 8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029 not found: ID does not exist" containerID="8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029" Nov 02 18:19:13.029692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.029628 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029"} err="failed to get container status \"8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029\": rpc error: code = NotFound desc = could not find container \"8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029\": container with ID starting with 8c7960ff07779fbb223d9f0d7b45599c27b5f31ffac9415984a726e12dd91029 not found: ID does not exist" Nov 02 18:19:13.172567 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.172522 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d"] Nov 02 18:19:13.178459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:13.178428 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-3dc055-openshift-default-846886d5-n5k5d"] Nov 02 18:19:15.081764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:15.081711 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" path="/var/lib/kubelet/pods/289c4ea7-7005-4b14-92c3-f963d3a7242e/volumes" Nov 02 18:19:21.049089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.049050 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524"] Nov 02 18:19:21.049931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.049906 5647 cpu_manager.go:414] "containerMap: removing container" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" Nov 02 18:19:21.050069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.049933 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" Nov 02 18:19:21.050069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.049956 5647 cpu_manager.go:414] "containerMap: removing container" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" Nov 02 18:19:21.050069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.049963 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" Nov 02 18:19:21.050237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.050106 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" Nov 02 18:19:21.050237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.050121 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="289c4ea7-7005-4b14-92c3-f963d3a7242e" containerName="istio-proxy" Nov 02 18:19:21.055433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.055408 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.164868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.164823 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a8da6e-serving-cert\"" Nov 02 18:19:21.170850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.170817 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524"] Nov 02 18:19:21.201000 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.200966 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.201183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.201088 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.201183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.201169 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbwh\" (UniqueName: \"kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.201298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.201229 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.322432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.322291 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.322432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.322394 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.322711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.322471 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbwh\" (UniqueName: \"kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.322711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.322533 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.324592 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.324560 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.327865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.327836 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.331692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.331668 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.384157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.384122 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbwh\" (UniqueName: \"kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh\") pod \"maas-api-e2e-extmod-a-a8da6e-697b544656-7w524\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:21.669331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:21.669297 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:22.052107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:22.052079 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524"] Nov 02 18:19:22.053015 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:19:22.052987 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ee3dfb_5cfc_45fc_9ad1_6a41d809e040.slice/crio-b501bb597fd89b4715223c28149426c7c9e9207e83ba0a9bac0a0fec7bea4564 WatchSource:0}: Error finding container b501bb597fd89b4715223c28149426c7c9e9207e83ba0a9bac0a0fec7bea4564: Status 404 returned error can't find the container with id b501bb597fd89b4715223c28149426c7c9e9207e83ba0a9bac0a0fec7bea4564 Nov 02 18:19:22.063670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:22.063632 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" event={"ID":"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040","Type":"ContainerStarted","Data":"b501bb597fd89b4715223c28149426c7c9e9207e83ba0a9bac0a0fec7bea4564"} Nov 02 18:19:23.071482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:23.071440 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" event={"ID":"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040","Type":"ContainerStarted","Data":"7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8"} Nov 02 18:19:23.071949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:23.071655 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:23.217194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:23.217135 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" podStartSLOduration=1.29881787 podStartE2EDuration="2.217120754s" totalImagesPullingTime="918.302884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:23.21528208 +0000 UTC m=+3350.288809285" watchObservedRunningTime="2026-11-02 18:19:23.217120754 +0000 UTC m=+3350.290647956" Nov 02 18:19:29.110938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:29.110909 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:19:35.951946 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:19:35.951118 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:19:59.671947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.671910 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2"] Nov 02 18:19:59.677566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.677546 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.726603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.726569 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.726791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.726627 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.726841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.726816 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m42r5\" (UniqueName: \"kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.726968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.726955 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.783327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.783282 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-cd738b-serving-cert\"" Nov 02 18:19:59.786000 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.785977 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2"] Nov 02 18:19:59.828211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.828172 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m42r5\" (UniqueName: \"kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.828424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.828354 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.828424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.828390 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.828572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.828435 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.828572 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:19:59.828509 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-serving-cert: secret "maas-api-e2e-extmod-a-cd738b-serving-cert" not found Nov 02 18:19:59.828572 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:19:59.828568 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls podName:0ad3399d-fc04-4ceb-884f-588eb7e7ad96 nodeName:}" failed. No retries permitted until 2026-11-02 18:20:00.328552518 +0000 UTC m=+3387.402079713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls") pod "maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" (UID: "0ad3399d-fc04-4ceb-884f-588eb7e7ad96") : secret "maas-api-e2e-extmod-a-cd738b-serving-cert" not found Nov 02 18:19:59.829221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.829196 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.831540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.831514 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:19:59.890366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:19:59.890328 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m42r5\" (UniqueName: \"kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:20:00.344840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:00.344804 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:20:00.347504 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:00.347481 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\") pod \"maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:20:00.591042 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:00.591004 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:20:00.970407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:00.970380 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2"] Nov 02 18:20:00.972549 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:20:00.972522 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad3399d_fc04_4ceb_884f_588eb7e7ad96.slice/crio-a310116b7d3bab003c6a92411d987a8ad30cbc4669a6e91b86af4f8634ed9f60 WatchSource:0}: Error finding container a310116b7d3bab003c6a92411d987a8ad30cbc4669a6e91b86af4f8634ed9f60: Status 404 returned error can't find the container with id a310116b7d3bab003c6a92411d987a8ad30cbc4669a6e91b86af4f8634ed9f60 Nov 02 18:20:01.333453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:01.333368 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" event={"ID":"0ad3399d-fc04-4ceb-884f-588eb7e7ad96","Type":"ContainerStarted","Data":"a310116b7d3bab003c6a92411d987a8ad30cbc4669a6e91b86af4f8634ed9f60"} Nov 02 18:20:02.341796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:02.341761 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" event={"ID":"0ad3399d-fc04-4ceb-884f-588eb7e7ad96","Type":"ContainerStarted","Data":"e63147d8b06a4bd49207504134c46ad6ab87ebf891f2046b16767c342402b3f5"} Nov 02 18:20:02.342272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:02.341840 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:20:02.460992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:02.460928 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" podStartSLOduration=2.377284963 podStartE2EDuration="3.460913117s" totalImagesPullingTime="1.083628154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:02.459177576 +0000 UTC m=+3389.532704816" watchObservedRunningTime="2026-11-02 18:20:02.460913117 +0000 UTC m=+3389.534440321" Nov 02 18:20:05.949767 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:20:05.949708 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:20:08.353303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:08.353271 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:20:09.220461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:09.220428 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:20:14.056621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:14.056577 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p"] Nov 02 18:20:19.057148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.057074 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" podUID="acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" containerName="payload-pre-processing" containerID="cri-o://9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6" gracePeriod=25 Nov 02 18:20:19.337093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.337058 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:20:19.427340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.427301 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume\") pod \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " Nov 02 18:20:19.427533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.427416 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z\" (UniqueName: \"kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z\") pod \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\" (UID: \"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd\") " Nov 02 18:20:19.427706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.427682 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume" pod "acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" (UID: "acfd12b6-aaa2-494c-b6c8-9aa0968b87dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:20:19.429809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.429779 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z" pod "acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" (UID: "acfd12b6-aaa2-494c-b6c8-9aa0968b87dd"). InnerVolumeSpecName "kube-api-access-24c7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:20:19.522902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.522856 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:20:19.523082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.522912 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:20:19.529441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.529400 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:20:19.529441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.529438 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24c7z\" (UniqueName: \"kubernetes.io/projected/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd-kube-api-access-24c7z\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:20:19.565073 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.564357 5647 generic.go:356] "Generic (PLEG): container finished" podID="acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" containerID="9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6" exitCode=0 Nov 02 18:20:19.565340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.564450 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" event={"ID":"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd","Type":"ContainerDied","Data":"9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6"} Nov 02 18:20:19.565537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.564492 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" Nov 02 18:20:19.565537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.565390 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p" event={"ID":"acfd12b6-aaa2-494c-b6c8-9aa0968b87dd","Type":"ContainerDied","Data":"63372cb1b64ceffedccdd39001d7d6fb30d843b134e16ae9ad61c49b2a6ef638"} Nov 02 18:20:19.565537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.565421 5647 scope.go:122] "RemoveContainer" containerID="9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6" Nov 02 18:20:19.577294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.577269 5647 scope.go:122] "RemoveContainer" containerID="9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6" Nov 02 18:20:19.577650 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:20:19.577622 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6\": container with ID starting with 9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6 not found: ID does not exist" containerID="9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6" Nov 02 18:20:19.577779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.577664 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6"} err="failed to get container status \"9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6\": rpc error: code = NotFound desc = could not find container \"9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6\": container with ID starting with 9c270e1201b51ca8346c04867cb8407c52db2edd8bf2aae79c10c790fb58d1b6 not found: ID does not exist" Nov 02 18:20:19.735993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.735959 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p"] Nov 02 18:20:19.740091 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:19.740050 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eb709f-79f5694df9-dnk6p"] Nov 02 18:20:20.575356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:20.575325 5647 generic.go:356] "Generic (PLEG): container finished" podID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerID="244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329" exitCode=0 Nov 02 18:20:20.575980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:20.575394 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" event={"ID":"f9166951-3f12-47ff-b3a9-75c1d616f309","Type":"ContainerDied","Data":"244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329"} Nov 02 18:20:20.575980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:20.575850 5647 scope.go:122] "RemoveContainer" containerID="244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329" Nov 02 18:20:20.577614 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:20.577583 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:20:20.577770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:20.577667 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:20:20.577856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:20.577783 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:20:21.082164 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:21.082133 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" path="/var/lib/kubelet/pods/acfd12b6-aaa2-494c-b6c8-9aa0968b87dd/volumes" Nov 02 18:20:21.584288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:21.584254 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" event={"ID":"f9166951-3f12-47ff-b3a9-75c1d616f309","Type":"ContainerStarted","Data":"231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d"} Nov 02 18:20:22.122250 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:22.122213 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:20:22.122679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:22.122583 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 02 18:20:22.122911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:22.122885 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 02 18:20:23.124759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:23.123852 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 02 18:20:23.124759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:23.123920 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 02 18:20:24.125000 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:24.124961 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 02 18:20:24.125655 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:24.125612 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 02 18:20:25.124761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:25.120202 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 02 18:20:25.124761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:25.120284 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 02 18:20:26.121211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:26.121164 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 02 18:20:26.122070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:26.122034 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 02 18:20:27.125478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:27.125447 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:20:27.125933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:27.125852 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:20:27.126565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:27.126547 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:20:35.948557 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:20:35.948516 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:20:42.547193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:20:42.547105 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:21:05.992956 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:21:05.992909 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:21:29.553644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.553602 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24"] Nov 02 18:21:29.554572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.554492 5647 cpu_manager.go:414] "containerMap: removing container" podUID="acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" containerName="payload-pre-processing" Nov 02 18:21:29.554572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.554525 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" containerName="payload-pre-processing" Nov 02 18:21:29.554922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.554684 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfd12b6-aaa2-494c-b6c8-9aa0968b87dd" containerName="payload-pre-processing" Nov 02 18:21:29.559016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.558993 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.638758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.618769 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a72f55-openshift-default-dockercfg-nz7ln\"" Nov 02 18:21:29.650523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.627505 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.684859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.682776 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24"] Nov 02 18:21:29.723228 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.720937 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.723861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.723823 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnftg\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.724131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.724113 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.724261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.724246 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.724412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.724398 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.724551 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.724536 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.724683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.724669 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.724828 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.724814 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.825974 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826051 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826084 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnftg\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826130 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826253 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826325 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826377 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826416 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826422 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826430 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826455 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826682 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826890 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.826939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.826894 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.828979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.828948 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.829152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.829134 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.885240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.885205 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:29.885436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:29.885335 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnftg\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg\") pod \"e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:30.175779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:30.175711 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:30.606949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:30.606925 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24"] Nov 02 18:21:30.608494 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:21:30.608469 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8df77cd_7619_4d45_ad37_7946076ed686.slice/crio-fae2c735fac819a47c8f382817252ef6cf6f8ac7c5b7d405b6921f6ed5c5afed WatchSource:0}: Error finding container fae2c735fac819a47c8f382817252ef6cf6f8ac7c5b7d405b6921f6ed5c5afed: Status 404 returned error can't find the container with id fae2c735fac819a47c8f382817252ef6cf6f8ac7c5b7d405b6921f6ed5c5afed Nov 02 18:21:30.611222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:30.611192 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:30.611311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:30.611257 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:30.611311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:30.611285 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:31.145470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:31.145428 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" event={"ID":"e8df77cd-7619-4d45-ad37-7946076ed686","Type":"ContainerStarted","Data":"15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8"} Nov 02 18:21:31.145670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:31.145498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" event={"ID":"e8df77cd-7619-4d45-ad37-7946076ed686","Type":"ContainerStarted","Data":"fae2c735fac819a47c8f382817252ef6cf6f8ac7c5b7d405b6921f6ed5c5afed"} Nov 02 18:21:31.176933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:31.176898 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:31.182872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:31.182848 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:31.371558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:31.371498 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podStartSLOduration=2.371482916 podStartE2EDuration="2.371482916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:31.319233781 +0000 UTC m=+3478.392760984" watchObservedRunningTime="2026-11-02 18:21:31.371482916 +0000 UTC m=+3478.445010113" Nov 02 18:21:32.152433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:32.152402 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:32.153518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:32.153496 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:21:33.348757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.343452 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m"] Nov 02 18:21:33.348757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.344093 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" podUID="9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" containerName="maas-api" containerID="cri-o://c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b" gracePeriod=30 Nov 02 18:21:33.749217 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.749177 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:21:33.776144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.768157 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca\") pod \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " Nov 02 18:21:33.776144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.768236 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w\" (UniqueName: \"kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w\") pod \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " Nov 02 18:21:33.776144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.768464 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\") pod \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " Nov 02 18:21:33.776144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.768593 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs\") pod \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\" (UID: \"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9\") " Nov 02 18:21:33.777136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.777098 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca" pod "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" (UID: "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:33.816902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.815107 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w" pod "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" (UID: "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9"). InnerVolumeSpecName "kube-api-access-9dp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:33.819535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.818982 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls" pod "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" (UID: "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:21:33.819535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.819074 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs" pod "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" (UID: "9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:21:33.877156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.877113 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:33.878294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.878262 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:33.878516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.878499 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:33.878634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:33.878620 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dp5w\" (UniqueName: \"kubernetes.io/projected/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9-kube-api-access-9dp5w\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:34.165665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.165628 5647 generic.go:356] "Generic (PLEG): container finished" podID="9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" containerID="c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b" exitCode=0 Nov 02 18:21:34.165908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.165696 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" Nov 02 18:21:34.165908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.165713 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" event={"ID":"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9","Type":"ContainerDied","Data":"c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b"} Nov 02 18:21:34.165908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.165816 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m" event={"ID":"9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9","Type":"ContainerDied","Data":"d36c3e24e383c69e31527e14c84767ce13f160692335f88d48231b5b96cb6cc2"} Nov 02 18:21:34.165908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.165847 5647 scope.go:122] "RemoveContainer" containerID="c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b" Nov 02 18:21:34.177174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.177147 5647 scope.go:122] "RemoveContainer" containerID="c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b" Nov 02 18:21:34.177575 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:21:34.177550 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b\": container with ID starting with c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b not found: ID does not exist" containerID="c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b" Nov 02 18:21:34.177694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.177583 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b"} err="failed to get container status \"c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b\": rpc error: code = NotFound desc = could not find container \"c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b\": container with ID starting with c58e9f3406ea3cb527768fa5aa1c558c5cffb8e2cdf2fdc8654d6c5c11ddb24b not found: ID does not exist" Nov 02 18:21:34.334190 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.334159 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m"] Nov 02 18:21:34.406526 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:34.406448 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-eb709f-6fc7f4b54d-ff44m"] Nov 02 18:21:35.083849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:35.083817 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" path="/var/lib/kubelet/pods/9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9/volumes" Nov 02 18:21:35.957927 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:21:35.957885 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:21:36.006951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:36.006916 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf"] Nov 02 18:21:36.007167 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:36.007140 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" containerID="cri-o://053cd1823c7f8b826be377b8ccc7f8470a714443dc4968e4b79f0fbfbc30d999" gracePeriod=30 Nov 02 18:21:39.093638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.093602 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc"] Nov 02 18:21:39.094349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.094324 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" containerName="maas-api" Nov 02 18:21:39.094510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.094347 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" containerName="maas-api" Nov 02 18:21:39.094583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.094560 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5d8ca4-b88f-4bd6-a767-3d8c045dfef9" containerName="maas-api" Nov 02 18:21:39.099643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.099619 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.162861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.162829 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc"] Nov 02 18:21:39.210595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.210567 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-cd9ae6-openshift-default-dockercfg-9tp6x\"" Nov 02 18:21:39.211625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.211582 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.211632 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.211789 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.211969 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8j59\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.212008 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.212046 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.212120 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.212186 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.215761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.212232 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.313808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.313772 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314009 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.313894 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8j59\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314009 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.313924 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314009 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.313942 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314013 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314071 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314119 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314163 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314189 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314517 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314559 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314653 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.314889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.314866 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.315041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.315018 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.316445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.316416 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.316605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.316587 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.387350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.387271 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8j59\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.391040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.391011 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token\") pod \"e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.431127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.431096 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:39.903482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.903429 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc"] Nov 02 18:21:39.930721 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:21:39.930687 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd35a2c9_eccb_4d26_bcb0_22628179fda8.slice/crio-cbdf498c64676320c413eb1104aef07fba3233a7baa65a2400c96cdf9b7b4bc2 WatchSource:0}: Error finding container cbdf498c64676320c413eb1104aef07fba3233a7baa65a2400c96cdf9b7b4bc2: Status 404 returned error can't find the container with id cbdf498c64676320c413eb1104aef07fba3233a7baa65a2400c96cdf9b7b4bc2 Nov 02 18:21:39.933402 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.933357 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:39.933542 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.933455 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:39.933542 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:39.933509 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:40.209778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:40.209608 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" event={"ID":"cd35a2c9-eccb-4d26-bcb0-22628179fda8","Type":"ContainerStarted","Data":"3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5"} Nov 02 18:21:40.209778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:40.209707 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" event={"ID":"cd35a2c9-eccb-4d26-bcb0-22628179fda8","Type":"ContainerStarted","Data":"cbdf498c64676320c413eb1104aef07fba3233a7baa65a2400c96cdf9b7b4bc2"} Nov 02 18:21:40.336530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:40.336457 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podStartSLOduration=1.33643607 podStartE2EDuration="1.33643607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:40.334322286 +0000 UTC m=+3487.407849490" watchObservedRunningTime="2026-11-02 18:21:40.33643607 +0000 UTC m=+3487.409963275" Nov 02 18:21:40.431997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:40.431966 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:41.221707 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.219836 5647 generic.go:356] "Generic (PLEG): container finished" podID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerID="053cd1823c7f8b826be377b8ccc7f8470a714443dc4968e4b79f0fbfbc30d999" exitCode=0 Nov 02 18:21:41.221707 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.219880 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" event={"ID":"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05","Type":"ContainerDied","Data":"053cd1823c7f8b826be377b8ccc7f8470a714443dc4968e4b79f0fbfbc30d999"} Nov 02 18:21:41.221707 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.219958 5647 scope.go:122] "RemoveContainer" containerID="1a43bdfc771d0d5d1893f35c7f949a08f68255825657e1208a21e130de09f606" Nov 02 18:21:41.271678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.271651 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:21:41.349023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.348987 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349068 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349160 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349205 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349350 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349382 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349434 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349464 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.349543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349511 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8\") pod \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\" (UID: \"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05\") " Nov 02 18:21:41.351361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.349697 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:41.351361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.350079 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.351361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.350773 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:41.351361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.351023 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:41.351361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.351058 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:21:41.351361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.351131 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:41.352554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.352520 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "kube-api-access-77ft8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:41.353044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.353007 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:21:41.353806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.353705 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:21:41.353806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.353784 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo" pod "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" (UID: "1ecb0add-9f6e-478f-a1dd-0e5524c4bf05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454095 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454134 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454153 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77ft8\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-kube-api-access-77ft8\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454166 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454180 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454193 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454204 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.455231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.454217 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:21:41.471123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.470895 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:41.569628 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.569590 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss"] Nov 02 18:21:41.570263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.570248 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" Nov 02 18:21:41.570321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.570265 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" Nov 02 18:21:41.570321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.570279 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" Nov 02 18:21:41.570321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.570285 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" Nov 02 18:21:41.570463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.570410 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" Nov 02 18:21:41.570463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.570421 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" containerName="istio-proxy" Nov 02 18:21:41.574142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.574122 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.629908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.629824 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-24a8b0-openshift-default-dockercfg-w8d4s\"" Nov 02 18:21:41.636554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.636520 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss"] Nov 02 18:21:41.657017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.656987 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657020 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657167 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvwr\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657210 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657325 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657366 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657522 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.657610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.657597 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760367 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvwr\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760424 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760495 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760537 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760622 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760651 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.760761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760683 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.761356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760823 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.761356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.760851 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.762954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.762764 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.763113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.763037 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.764208 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.764085 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.764929 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.764489 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.765685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.765609 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.766023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.765873 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.766781 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.766759 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.823612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.823582 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.823933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.823908 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvwr\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr\") pod \"e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:41.890154 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:41.890077 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:42.227311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.227224 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" Nov 02 18:21:42.227311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.227265 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf" event={"ID":"1ecb0add-9f6e-478f-a1dd-0e5524c4bf05","Type":"ContainerDied","Data":"88795268101ca5a363aabfdb13d91821a701f0dfc7cf2f831459e7005b47ac1b"} Nov 02 18:21:42.227861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.227332 5647 scope.go:122] "RemoveContainer" containerID="053cd1823c7f8b826be377b8ccc7f8470a714443dc4968e4b79f0fbfbc30d999" Nov 02 18:21:42.227861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.227699 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:42.228823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.228807 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:21:42.323875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.323850 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss"] Nov 02 18:21:42.325474 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:21:42.325424 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92354e2e_707d_4046_92d3_8468eca9b7eb.slice/crio-a79b83f9e8504970255112129bb85c5c701ccc93ac1eaa8f8945e5e597a4b4a4 WatchSource:0}: Error finding container a79b83f9e8504970255112129bb85c5c701ccc93ac1eaa8f8945e5e597a4b4a4: Status 404 returned error can't find the container with id a79b83f9e8504970255112129bb85c5c701ccc93ac1eaa8f8945e5e597a4b4a4 Nov 02 18:21:42.328815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.328782 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:42.328903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.328848 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:42.328903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.328881 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:21:42.527688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.527581 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf"] Nov 02 18:21:42.530301 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:42.530274 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-eb709f-openshift-default-64b5568cd5-8khqf"] Nov 02 18:21:43.081754 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:43.081713 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ecb0add-9f6e-478f-a1dd-0e5524c4bf05" path="/var/lib/kubelet/pods/1ecb0add-9f6e-478f-a1dd-0e5524c4bf05/volumes" Nov 02 18:21:43.235791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:43.235750 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" event={"ID":"92354e2e-707d-4046-92d3-8468eca9b7eb","Type":"ContainerStarted","Data":"87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe"} Nov 02 18:21:43.236185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:43.235802 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" event={"ID":"92354e2e-707d-4046-92d3-8468eca9b7eb","Type":"ContainerStarted","Data":"a79b83f9e8504970255112129bb85c5c701ccc93ac1eaa8f8945e5e597a4b4a4"} Nov 02 18:21:43.355119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:43.355071 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podStartSLOduration=2.3550589840000002 podStartE2EDuration="2.355058984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:43.354452938 +0000 UTC m=+3490.427980142" watchObservedRunningTime="2026-11-02 18:21:43.355058984 +0000 UTC m=+3490.428586247" Nov 02 18:21:43.890471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:43.890439 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:43.897029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:43.897004 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:44.242507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:44.242403 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:44.243356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:44.243333 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:21:46.528386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:46.528355 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:21:52.521105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.521070 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg"] Nov 02 18:21:52.525992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.525974 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.626345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.626290 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9xb6\" (UniqueName: \"kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6\") pod \"payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.626769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.626717 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.642502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.642473 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-24a8b0\"" Nov 02 18:21:52.642816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.642790 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-24a8b0-dockercfg-w2m6g\"" Nov 02 18:21:52.687755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.679344 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg"] Nov 02 18:21:52.687755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.680420 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f"] Nov 02 18:21:52.690681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.690657 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:52.729802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.728663 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xb6\" (UniqueName: \"kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6\") pod \"payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.729802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.728754 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.732757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.730443 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.735137 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.735103 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f"] Nov 02 18:21:52.789382 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.789315 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xb6\" (UniqueName: \"kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6\") pod \"payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.830665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.830633 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9ks\" (UniqueName: \"kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks\") pod \"payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:52.830869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.830791 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:52.839321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.839295 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:52.934697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.934655 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:52.934945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.934920 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9ks\" (UniqueName: \"kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks\") pod \"payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:52.935506 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.935486 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:52.992894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:52.992860 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9ks\" (UniqueName: \"kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks\") pod \"payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:53.022351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.022315 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:53.216936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.216912 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg"] Nov 02 18:21:53.218397 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:21:53.218360 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adc40e5_3f98_4f65_aa7b_5b76484d147b.slice/crio-f9b3f7b52e4069dc4e0d3a818f6cbb78a37f924363ac4be1daf1b8d0d4318ff9 WatchSource:0}: Error finding container f9b3f7b52e4069dc4e0d3a818f6cbb78a37f924363ac4be1daf1b8d0d4318ff9: Status 404 returned error can't find the container with id f9b3f7b52e4069dc4e0d3a818f6cbb78a37f924363ac4be1daf1b8d0d4318ff9 Nov 02 18:21:53.339709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.339670 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" event={"ID":"4adc40e5-3f98-4f65-aa7b-5b76484d147b","Type":"ContainerStarted","Data":"53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d"} Nov 02 18:21:53.339919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.339724 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" event={"ID":"4adc40e5-3f98-4f65-aa7b-5b76484d147b","Type":"ContainerStarted","Data":"f9b3f7b52e4069dc4e0d3a818f6cbb78a37f924363ac4be1daf1b8d0d4318ff9"} Nov 02 18:21:53.339919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.339814 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:21:53.405864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.405833 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f"] Nov 02 18:21:53.406884 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:21:53.406858 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f03b84_aeda_4390_9f60_e26de17a455a.slice/crio-f1f1324b9eacfcd34215eedb6352a3fd889320384813eda64c842b1ecac3127a WatchSource:0}: Error finding container f1f1324b9eacfcd34215eedb6352a3fd889320384813eda64c842b1ecac3127a: Status 404 returned error can't find the container with id f1f1324b9eacfcd34215eedb6352a3fd889320384813eda64c842b1ecac3127a Nov 02 18:21:53.460529 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:53.456519 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" podStartSLOduration=1.4565034 podStartE2EDuration="1.4565034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:53.454878124 +0000 UTC m=+3500.528405326" watchObservedRunningTime="2026-11-02 18:21:53.4565034 +0000 UTC m=+3500.530030604" Nov 02 18:21:54.348877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:54.348842 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" event={"ID":"18f03b84-aeda-4390-9f60-e26de17a455a","Type":"ContainerStarted","Data":"29843a2588d3f53290714452de88247632cd140b0f794e358066c051e383bd7e"} Nov 02 18:21:54.349320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:54.348896 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" event={"ID":"18f03b84-aeda-4390-9f60-e26de17a455a","Type":"ContainerStarted","Data":"f1f1324b9eacfcd34215eedb6352a3fd889320384813eda64c842b1ecac3127a"} Nov 02 18:21:54.349320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:54.349045 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:21:54.466387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:21:54.466338 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" podStartSLOduration=2.466325252 podStartE2EDuration="2.466325252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:54.46502488 +0000 UTC m=+3501.538552094" watchObservedRunningTime="2026-11-02 18:21:54.466325252 +0000 UTC m=+3501.539852481" Nov 02 18:22:04.353183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:22:04.353150 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:22:05.358352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:22:05.358323 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:22:05.947280 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:22:05.947247 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:22:35.948142 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:22:35.948102 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:22:44.094969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:22:44.094927 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:23:05.947709 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:23:05.947668 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:23:06.670157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.670117 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl"] Nov 02 18:23:06.674359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.674329 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.778475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.778439 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.778475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.778484 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.778708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.778505 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.778793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.778701 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b792m\" (UniqueName: \"kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.779282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.779264 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a72f55-serving-cert\"" Nov 02 18:23:06.782383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.782359 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl"] Nov 02 18:23:06.879790 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.879759 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b792m\" (UniqueName: \"kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.879992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.879873 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.879992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.879901 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.879992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.879922 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.880155 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:23:06.880065 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-serving-cert: secret "maas-api-e2e-extmod-b-a72f55-serving-cert" not found Nov 02 18:23:06.880209 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:23:06.880155 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls podName:004cb34f-80f6-44fa-bc55-6c6e4341bd27 nodeName:}" failed. No retries permitted until 2026-11-02 18:23:07.380126342 +0000 UTC m=+3574.453653525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls") pod "maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" (UID: "004cb34f-80f6-44fa-bc55-6c6e4341bd27") : secret "maas-api-e2e-extmod-b-a72f55-serving-cert" not found Nov 02 18:23:06.880661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.880564 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.882840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.882803 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:06.941393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:06.941325 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b792m\" (UniqueName: \"kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:07.386333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:07.386296 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:07.388996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:07.388970 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:07.588443 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:07.588410 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:07.973156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:07.973125 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl"] Nov 02 18:23:07.974573 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:23:07.974534 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004cb34f_80f6_44fa_bc55_6c6e4341bd27.slice/crio-0e2af5d5c2015210dd4cd9534ae91ffb29b0d3e4a9d2f71d7995a1224287ca36 WatchSource:0}: Error finding container 0e2af5d5c2015210dd4cd9534ae91ffb29b0d3e4a9d2f71d7995a1224287ca36: Status 404 returned error can't find the container with id 0e2af5d5c2015210dd4cd9534ae91ffb29b0d3e4a9d2f71d7995a1224287ca36 Nov 02 18:23:07.976097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:07.976075 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:23:08.825178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:08.825146 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" event={"ID":"004cb34f-80f6-44fa-bc55-6c6e4341bd27","Type":"ContainerStarted","Data":"0e2af5d5c2015210dd4cd9534ae91ffb29b0d3e4a9d2f71d7995a1224287ca36"} Nov 02 18:23:09.833304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:09.833262 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" event={"ID":"004cb34f-80f6-44fa-bc55-6c6e4341bd27","Type":"ContainerStarted","Data":"f43b45144203a947de164c7d433966b4df864bb04a22cc7dc5ca420ab943afd2"} Nov 02 18:23:09.833721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:09.833420 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:09.991751 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:09.991690 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" podStartSLOduration=3.205756599 podStartE2EDuration="3.991671563s" totalImagesPullingTime="785.914964ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:09.989609859 +0000 UTC m=+3577.063137086" watchObservedRunningTime="2026-11-02 18:23:09.991671563 +0000 UTC m=+3577.065198769" Nov 02 18:23:15.844692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:15.844658 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:23:16.417289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:16.417242 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:23:25.954654 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:25.954615 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql"] Nov 02 18:23:25.958987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:25.958968 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.064509 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.064469 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-24a8b0-serving-cert\"" Nov 02 18:23:26.067249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.067221 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.067249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.067257 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.067484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.067400 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql"] Nov 02 18:23:26.067546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.067528 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.067587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.067560 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldns\" (UniqueName: \"kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.169241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.169205 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.169241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.169244 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.169528 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:23:26.169364 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-serving-cert: secret "maas-api-e2e-extmod-a-24a8b0-serving-cert" not found Nov 02 18:23:26.169528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.169406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.169528 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:23:26.169436 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls podName:d5f437d6-10a2-4596-ae1f-db9478812213 nodeName:}" failed. No retries permitted until 2026-11-02 18:23:26.669419571 +0000 UTC m=+3593.742946756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls") pod "maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" (UID: "d5f437d6-10a2-4596-ae1f-db9478812213") : secret "maas-api-e2e-extmod-a-24a8b0-serving-cert" not found Nov 02 18:23:26.169528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.169454 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldns\" (UniqueName: \"kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.170167 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.170145 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.172028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.172010 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.241817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.241710 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldns\" (UniqueName: \"kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.677441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.677407 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.680044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.680013 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\") pod \"maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:26.872990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:26.872950 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:27.255550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:27.255519 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql"] Nov 02 18:23:27.257339 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:23:27.257311 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f437d6_10a2_4596_ae1f_db9478812213.slice/crio-74ab18548763d4797f48186ee2f24e547eb88ddfea8a2d13eaaa269f9e2bf2f2 WatchSource:0}: Error finding container 74ab18548763d4797f48186ee2f24e547eb88ddfea8a2d13eaaa269f9e2bf2f2: Status 404 returned error can't find the container with id 74ab18548763d4797f48186ee2f24e547eb88ddfea8a2d13eaaa269f9e2bf2f2 Nov 02 18:23:27.960624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:27.960588 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" event={"ID":"d5f437d6-10a2-4596-ae1f-db9478812213","Type":"ContainerStarted","Data":"74ab18548763d4797f48186ee2f24e547eb88ddfea8a2d13eaaa269f9e2bf2f2"} Nov 02 18:23:28.985760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:28.983129 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" event={"ID":"d5f437d6-10a2-4596-ae1f-db9478812213","Type":"ContainerStarted","Data":"c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02"} Nov 02 18:23:28.985760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:28.984367 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:29.103778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:29.103683 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" podStartSLOduration=3.251497384 podStartE2EDuration="4.103665143s" totalImagesPullingTime="852.167759ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:29.101904408 +0000 UTC m=+3596.175431612" watchObservedRunningTime="2026-11-02 18:23:29.103665143 +0000 UTC m=+3596.177192362" Nov 02 18:23:34.460816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:34.460781 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:23:34.466560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:34.466536 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:23:36.003949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:36.003922 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:23:36.020277 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:23:36.020238 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:23:48.193921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:48.193879 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:23:48.194524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:48.193995 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:23:51.141089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.141057 5647 generic.go:356] "Generic (PLEG): container finished" podID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerID="3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5" exitCode=0 Nov 02 18:23:51.141585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.141136 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" event={"ID":"cd35a2c9-eccb-4d26-bcb0-22628179fda8","Type":"ContainerDied","Data":"3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5"} Nov 02 18:23:51.141642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.141593 5647 scope.go:122] "RemoveContainer" containerID="3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5" Nov 02 18:23:51.142949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.142921 5647 generic.go:356] "Generic (PLEG): container finished" podID="e8df77cd-7619-4d45-ad37-7946076ed686" containerID="15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8" exitCode=0 Nov 02 18:23:51.143035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.142961 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" event={"ID":"e8df77cd-7619-4d45-ad37-7946076ed686","Type":"ContainerDied","Data":"15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8"} Nov 02 18:23:51.143369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.143353 5647 scope.go:122] "RemoveContainer" containerID="15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8" Nov 02 18:23:51.143702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.143670 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:23:51.143814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.143789 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:23:51.143972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.143843 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:23:51.145034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.145004 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:23:51.145132 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.145070 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:23:51.145132 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:51.145105 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:23:52.160288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.160249 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" event={"ID":"cd35a2c9-eccb-4d26-bcb0-22628179fda8","Type":"ContainerStarted","Data":"a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d"} Nov 02 18:23:52.162532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.162508 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" event={"ID":"e8df77cd-7619-4d45-ad37-7946076ed686","Type":"ContainerStarted","Data":"5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8"} Nov 02 18:23:52.176308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.176278 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:23:52.176564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.176502 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 02 18:23:52.176661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.176595 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 02 18:23:52.441764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.438778 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:23:52.441764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.438919 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 02 18:23:52.441764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:52.438974 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 02 18:23:53.176603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:53.176577 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 02 18:23:53.177050 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:53.176619 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 02 18:23:53.433918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:53.433785 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 02 18:23:53.434401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:53.434275 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 02 18:23:54.176388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:54.176353 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 02 18:23:54.176585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:54.176408 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 02 18:23:54.435262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:54.435160 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 02 18:23:54.435895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:54.435854 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 02 18:23:55.176387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:55.176280 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 02 18:23:55.176387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:55.176351 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 02 18:23:55.435239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:55.435141 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 02 18:23:55.435513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:55.435483 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 02 18:23:56.176224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:56.176184 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 02 18:23:56.176432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:56.176237 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 02 18:23:56.435418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:56.435312 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 02 18:23:56.435714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:56.435667 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 02 18:23:57.176700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:57.176665 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Nov 02 18:23:57.176959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:57.176716 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Nov 02 18:23:57.228761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:57.227846 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:23:57.434817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:57.434258 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body= Nov 02 18:23:57.434817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:57.434340 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" Nov 02 18:23:58.180980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:58.180941 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:23:58.181394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:58.181343 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:23:58.182197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:58.182176 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:23:58.437156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:58.437084 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:23:58.438232 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:23:58.438211 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:24:05.947542 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:24:05.947502 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:24:14.692760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:14.692703 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:24:25.397509 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:25.397476 5647 generic.go:356] "Generic (PLEG): container finished" podID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerID="87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe" exitCode=0 Nov 02 18:24:25.398083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:25.397557 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" event={"ID":"92354e2e-707d-4046-92d3-8468eca9b7eb","Type":"ContainerDied","Data":"87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe"} Nov 02 18:24:25.398083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:25.398022 5647 scope.go:122] "RemoveContainer" containerID="87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe" Nov 02 18:24:25.399763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:25.399687 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:24:25.399886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:25.399802 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:24:25.399886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:25.399839 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:24:26.428084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:26.428037 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" event={"ID":"92354e2e-707d-4046-92d3-8468eca9b7eb","Type":"ContainerStarted","Data":"668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800"} Nov 02 18:24:26.891134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:26.891089 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:24:26.892644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:26.892607 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 02 18:24:26.892806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:26.892677 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 02 18:24:27.891352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:27.891318 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 02 18:24:27.891790 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:27.891401 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 02 18:24:28.890988 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:28.890949 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 02 18:24:28.891186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:28.891021 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 02 18:24:29.242830 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:29.242750 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:24:29.891194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:29.891153 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 02 18:24:29.891418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:29.891243 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 02 18:24:30.890798 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:30.890757 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Nov 02 18:24:30.891252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:30.890820 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Nov 02 18:24:31.895884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:31.895847 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:24:31.896894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:31.896872 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:24:33.374666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.374630 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d"] Nov 02 18:24:33.377935 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.377916 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.431531 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.431500 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e4b90c-openshift-default-dockercfg-6kx9n\"" Nov 02 18:24:33.459583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459553 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.459771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459601 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8q8\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.459771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459663 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.459771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459704 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.459913 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459800 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.459913 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459882 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.459996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.459983 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.460040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.460029 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.460079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.460054 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.560938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.560899 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.560950 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561131 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561198 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8q8\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561354 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561233 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561383 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561448 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561563 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561631 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561648 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561828 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561807 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561858 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.561945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.561906 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.562233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.562207 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.563945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.563914 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.564120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.564104 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.580562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.580530 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d"] Nov 02 18:24:33.620188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.620154 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.620425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.620200 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8q8\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:33.690701 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:33.690613 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:34.140472 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.140430 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d"] Nov 02 18:24:34.176693 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:24:34.176665 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9ae0d2_d57a_4afd_b54a_6344f64b5004.slice/crio-c27a559941c3e1ba1237735b7aa232244e9dd8ebf068a63ff1fee09285db03d5 WatchSource:0}: Error finding container c27a559941c3e1ba1237735b7aa232244e9dd8ebf068a63ff1fee09285db03d5: Status 404 returned error can't find the container with id c27a559941c3e1ba1237735b7aa232244e9dd8ebf068a63ff1fee09285db03d5 Nov 02 18:24:34.179974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.179929 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:24:34.180093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.180008 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:24:34.180093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.180036 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:24:34.487120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.487027 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" event={"ID":"6c9ae0d2-d57a-4afd-b54a-6344f64b5004","Type":"ContainerStarted","Data":"b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f"} Nov 02 18:24:34.487120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.487094 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" event={"ID":"6c9ae0d2-d57a-4afd-b54a-6344f64b5004","Type":"ContainerStarted","Data":"c27a559941c3e1ba1237735b7aa232244e9dd8ebf068a63ff1fee09285db03d5"} Nov 02 18:24:34.611932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.611869 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podStartSLOduration=1.611850251 podStartE2EDuration="1.611850251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:24:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:24:34.607206403 +0000 UTC m=+3661.680733605" watchObservedRunningTime="2026-11-02 18:24:34.611850251 +0000 UTC m=+3661.685377457" Nov 02 18:24:34.691234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:34.691205 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:35.693307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:35.693270 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 02 18:24:35.693702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:35.693323 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 02 18:24:35.952721 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:24:35.952637 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:24:36.691888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:36.691854 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 02 18:24:36.692072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:36.691963 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 02 18:24:37.691333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:37.691284 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 02 18:24:37.691784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:37.691356 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 02 18:24:38.691179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:38.691147 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 02 18:24:38.691399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:38.691202 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 02 18:24:39.691023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:39.690990 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Nov 02 18:24:39.691209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:39.691054 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Nov 02 18:24:40.695784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:40.695720 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:40.696253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:40.696152 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:40.696919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:40.696894 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:24:52.183592 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:52.183560 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh"] Nov 02 18:24:56.560249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:56.560214 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f"] Nov 02 18:24:56.676360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:56.676325 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg"] Nov 02 18:24:57.184815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.184683 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" podUID="72dc1ad5-e093-4a8b-830b-87632bd8b912" containerName="payload-processing" containerID="cri-o://d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976" gracePeriod=25 Nov 02 18:24:57.463273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.463245 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:24:57.557061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.557018 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume\") pod \"72dc1ad5-e093-4a8b-830b-87632bd8b912\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " Nov 02 18:24:57.557272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.557100 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz\" (UniqueName: \"kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz\") pod \"72dc1ad5-e093-4a8b-830b-87632bd8b912\" (UID: \"72dc1ad5-e093-4a8b-830b-87632bd8b912\") " Nov 02 18:24:57.557459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.557435 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume" pod "72dc1ad5-e093-4a8b-830b-87632bd8b912" (UID: "72dc1ad5-e093-4a8b-830b-87632bd8b912"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:24:57.557785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.557763 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc1ad5-e093-4a8b-830b-87632bd8b912-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:24:57.559417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.559399 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz" pod "72dc1ad5-e093-4a8b-830b-87632bd8b912" (UID: "72dc1ad5-e093-4a8b-830b-87632bd8b912"). InnerVolumeSpecName "kube-api-access-n72vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:24:57.659316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.659283 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72vz\" (UniqueName: \"kubernetes.io/projected/72dc1ad5-e093-4a8b-830b-87632bd8b912-kube-api-access-n72vz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:24:57.687934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.687897 5647 generic.go:356] "Generic (PLEG): container finished" podID="72dc1ad5-e093-4a8b-830b-87632bd8b912" containerID="d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976" exitCode=0 Nov 02 18:24:57.688139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.688010 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" Nov 02 18:24:57.688139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.688042 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" event={"ID":"72dc1ad5-e093-4a8b-830b-87632bd8b912","Type":"ContainerDied","Data":"d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976"} Nov 02 18:24:57.688139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.688101 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh" event={"ID":"72dc1ad5-e093-4a8b-830b-87632bd8b912","Type":"ContainerDied","Data":"e9a2c45173e1eb197f51fe3425c0b95330d8c76aa165637a69491600b104feec"} Nov 02 18:24:57.688139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.688127 5647 scope.go:122] "RemoveContainer" containerID="d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976" Nov 02 18:24:57.702184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.702159 5647 scope.go:122] "RemoveContainer" containerID="d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976" Nov 02 18:24:57.702478 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:24:57.702453 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976\": container with ID starting with d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976 not found: ID does not exist" containerID="d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976" Nov 02 18:24:57.702562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.702491 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976"} err="failed to get container status \"d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976\": rpc error: code = NotFound desc = could not find container \"d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976\": container with ID starting with d5d05efe68d50cb203dbcd3bcf0e9dd6dde26346cb4e5aca4cf6023a853ec976 not found: ID does not exist" Nov 02 18:24:57.856173 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.856102 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh"] Nov 02 18:24:57.859767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:57.859725 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cd738b-854f5cf8c7-6crhh"] Nov 02 18:24:59.082298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:24:59.082266 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72dc1ad5-e093-4a8b-830b-87632bd8b912" path="/var/lib/kubelet/pods/72dc1ad5-e093-4a8b-830b-87632bd8b912/volumes" Nov 02 18:25:01.560946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.560902 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" podUID="18f03b84-aeda-4390-9f60-e26de17a455a" containerName="payload-processing" containerID="cri-o://29843a2588d3f53290714452de88247632cd140b0f794e358066c051e383bd7e" gracePeriod=25 Nov 02 18:25:01.677579 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.677521 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" podUID="4adc40e5-3f98-4f65-aa7b-5b76484d147b" containerName="payload-pre-processing" containerID="cri-o://53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d" gracePeriod=25 Nov 02 18:25:01.717868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.717836 5647 generic.go:356] "Generic (PLEG): container finished" podID="18f03b84-aeda-4390-9f60-e26de17a455a" containerID="29843a2588d3f53290714452de88247632cd140b0f794e358066c051e383bd7e" exitCode=0 Nov 02 18:25:01.718078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.717903 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" event={"ID":"18f03b84-aeda-4390-9f60-e26de17a455a","Type":"ContainerDied","Data":"29843a2588d3f53290714452de88247632cd140b0f794e358066c051e383bd7e"} Nov 02 18:25:01.838080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.838046 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:25:01.928549 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.928516 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume\") pod \"18f03b84-aeda-4390-9f60-e26de17a455a\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " Nov 02 18:25:01.928779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.928695 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks\" (UniqueName: \"kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks\") pod \"18f03b84-aeda-4390-9f60-e26de17a455a\" (UID: \"18f03b84-aeda-4390-9f60-e26de17a455a\") " Nov 02 18:25:01.928993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.928967 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume" pod "18f03b84-aeda-4390-9f60-e26de17a455a" (UID: "18f03b84-aeda-4390-9f60-e26de17a455a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:01.929345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.929312 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18f03b84-aeda-4390-9f60-e26de17a455a-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:01.931063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.931041 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks" pod "18f03b84-aeda-4390-9f60-e26de17a455a" (UID: "18f03b84-aeda-4390-9f60-e26de17a455a"). InnerVolumeSpecName "kube-api-access-8g9ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:01.952160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:01.952126 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:25:02.030499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.030456 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume\") pod \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " Nov 02 18:25:02.030697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.030567 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6\" (UniqueName: \"kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6\") pod \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\" (UID: \"4adc40e5-3f98-4f65-aa7b-5b76484d147b\") " Nov 02 18:25:02.030906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.030884 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume" pod "4adc40e5-3f98-4f65-aa7b-5b76484d147b" (UID: "4adc40e5-3f98-4f65-aa7b-5b76484d147b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:02.031196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.031177 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9ks\" (UniqueName: \"kubernetes.io/projected/18f03b84-aeda-4390-9f60-e26de17a455a-kube-api-access-8g9ks\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:02.031241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.031201 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4adc40e5-3f98-4f65-aa7b-5b76484d147b-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:02.033012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.032990 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6" pod "4adc40e5-3f98-4f65-aa7b-5b76484d147b" (UID: "4adc40e5-3f98-4f65-aa7b-5b76484d147b"). InnerVolumeSpecName "kube-api-access-n9xb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:02.132421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.132377 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9xb6\" (UniqueName: \"kubernetes.io/projected/4adc40e5-3f98-4f65-aa7b-5b76484d147b-kube-api-access-n9xb6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:02.725874 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.725835 5647 generic.go:356] "Generic (PLEG): container finished" podID="4adc40e5-3f98-4f65-aa7b-5b76484d147b" containerID="53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d" exitCode=0 Nov 02 18:25:02.726352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.725886 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" Nov 02 18:25:02.726352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.725920 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" event={"ID":"4adc40e5-3f98-4f65-aa7b-5b76484d147b","Type":"ContainerDied","Data":"53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d"} Nov 02 18:25:02.726352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.725975 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg" event={"ID":"4adc40e5-3f98-4f65-aa7b-5b76484d147b","Type":"ContainerDied","Data":"f9b3f7b52e4069dc4e0d3a818f6cbb78a37f924363ac4be1daf1b8d0d4318ff9"} Nov 02 18:25:02.726352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.725996 5647 scope.go:122] "RemoveContainer" containerID="53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d" Nov 02 18:25:02.727597 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.727574 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" Nov 02 18:25:02.727773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.727593 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f" event={"ID":"18f03b84-aeda-4390-9f60-e26de17a455a","Type":"ContainerDied","Data":"f1f1324b9eacfcd34215eedb6352a3fd889320384813eda64c842b1ecac3127a"} Nov 02 18:25:02.741590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.741566 5647 scope.go:122] "RemoveContainer" containerID="53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d" Nov 02 18:25:02.742106 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:25:02.742075 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d\": container with ID starting with 53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d not found: ID does not exist" containerID="53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d" Nov 02 18:25:02.742232 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.742116 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d"} err="failed to get container status \"53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d\": rpc error: code = NotFound desc = could not find container \"53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d\": container with ID starting with 53cec2a26e1915ef5251aef73595b36a7c28e46fd48b094421eabc797037517d not found: ID does not exist" Nov 02 18:25:02.742232 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.742151 5647 scope.go:122] "RemoveContainer" containerID="29843a2588d3f53290714452de88247632cd140b0f794e358066c051e383bd7e" Nov 02 18:25:02.895994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.895956 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg"] Nov 02 18:25:02.946828 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:02.946790 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-24a8b0-7d96467c79-6mjwg"] Nov 02 18:25:03.009162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.009115 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f"] Nov 02 18:25:03.014622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.014594 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-24a8b0-dbf4747b5-9m94f"] Nov 02 18:25:03.082462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.082430 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f03b84-aeda-4390-9f60-e26de17a455a" path="/var/lib/kubelet/pods/18f03b84-aeda-4390-9f60-e26de17a455a/volumes" Nov 02 18:25:03.082906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.082890 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4adc40e5-3f98-4f65-aa7b-5b76484d147b" path="/var/lib/kubelet/pods/4adc40e5-3f98-4f65-aa7b-5b76484d147b/volumes" Nov 02 18:25:03.625006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.624970 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf"] Nov 02 18:25:03.625642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625625 5647 cpu_manager.go:414] "containerMap: removing container" podUID="18f03b84-aeda-4390-9f60-e26de17a455a" containerName="payload-processing" Nov 02 18:25:03.625689 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625646 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f03b84-aeda-4390-9f60-e26de17a455a" containerName="payload-processing" Nov 02 18:25:03.625689 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625675 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4adc40e5-3f98-4f65-aa7b-5b76484d147b" containerName="payload-pre-processing" Nov 02 18:25:03.625689 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625681 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4adc40e5-3f98-4f65-aa7b-5b76484d147b" containerName="payload-pre-processing" Nov 02 18:25:03.625689 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625690 5647 cpu_manager.go:414] "containerMap: removing container" podUID="72dc1ad5-e093-4a8b-830b-87632bd8b912" containerName="payload-processing" Nov 02 18:25:03.625857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625694 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dc1ad5-e093-4a8b-830b-87632bd8b912" containerName="payload-processing" Nov 02 18:25:03.625857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625809 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f03b84-aeda-4390-9f60-e26de17a455a" containerName="payload-processing" Nov 02 18:25:03.625857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625824 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4adc40e5-3f98-4f65-aa7b-5b76484d147b" containerName="payload-pre-processing" Nov 02 18:25:03.625857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.625829 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dc1ad5-e093-4a8b-830b-87632bd8b912" containerName="payload-processing" Nov 02 18:25:03.630783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.630763 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.738393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.738363 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf"] Nov 02 18:25:03.751799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.751771 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.751990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.751843 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.751990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.751930 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.752123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.752088 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.752188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.752142 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.752253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.752237 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.752343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.752330 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.752415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.752403 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.752454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.752426 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn8jg\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.853902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.853859 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.853918 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854027 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn8jg\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854112 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854169 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854195 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854281 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854295 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854324 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854371 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854518 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854615 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854693 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.854957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.854936 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.856625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.856595 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.857179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.857160 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.913950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.913883 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.914095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.914073 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn8jg\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg\") pod \"e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:03.946420 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:03.946387 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:04.385438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.385409 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf"] Nov 02 18:25:04.386434 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:25:04.386410 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0ce187_e98e_4042_b957_a6f1f53b92d8.slice/crio-0369ddb31cf08e33be93ef71ffacfece3cdd2f1c122975ec7d1c89659ba9927e WatchSource:0}: Error finding container 0369ddb31cf08e33be93ef71ffacfece3cdd2f1c122975ec7d1c89659ba9927e: Status 404 returned error can't find the container with id 0369ddb31cf08e33be93ef71ffacfece3cdd2f1c122975ec7d1c89659ba9927e Nov 02 18:25:04.389216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.389184 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:25:04.389278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.389253 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:25:04.389314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.389283 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:25:04.746922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.746820 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" event={"ID":"cb0ce187-e98e-4042-b957-a6f1f53b92d8","Type":"ContainerStarted","Data":"10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435"} Nov 02 18:25:04.746922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.746883 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" event={"ID":"cb0ce187-e98e-4042-b957-a6f1f53b92d8","Type":"ContainerStarted","Data":"0369ddb31cf08e33be93ef71ffacfece3cdd2f1c122975ec7d1c89659ba9927e"} Nov 02 18:25:04.947034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:04.946993 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:05.948335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:05.948299 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 02 18:25:05.948817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:05.948359 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 02 18:25:05.956170 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:25:05.956140 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:25:06.947425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:06.947383 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 02 18:25:06.947642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:06.947469 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 02 18:25:07.947037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:07.947001 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 02 18:25:07.947460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:07.947068 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 02 18:25:08.947717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:08.947676 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 02 18:25:08.948178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:08.947795 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 02 18:25:09.947683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:09.947645 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 02 18:25:09.947902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:09.947713 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 02 18:25:10.952659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:10.952631 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:10.953196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:10.952920 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:10.953640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:10.953623 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:25:11.130408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:11.130329 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podStartSLOduration=8.130311659 podStartE2EDuration="8.130311659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:04.869057713 +0000 UTC m=+3691.942584915" watchObservedRunningTime="2026-11-02 18:25:11.130311659 +0000 UTC m=+3698.203838866" Nov 02 18:25:15.009825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.009785 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql"] Nov 02 18:25:15.010860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.010823 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" podUID="d5f437d6-10a2-4596-ae1f-db9478812213" containerName="maas-api" containerID="cri-o://c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02" gracePeriod=30 Nov 02 18:25:15.485835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.485801 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:25:15.565470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.565381 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\") pod \"d5f437d6-10a2-4596-ae1f-db9478812213\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " Nov 02 18:25:15.565470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.565445 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca\") pod \"d5f437d6-10a2-4596-ae1f-db9478812213\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " Nov 02 18:25:15.565719 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.565598 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns\" (UniqueName: \"kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns\") pod \"d5f437d6-10a2-4596-ae1f-db9478812213\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " Nov 02 18:25:15.565719 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.565686 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs\") pod \"d5f437d6-10a2-4596-ae1f-db9478812213\" (UID: \"d5f437d6-10a2-4596-ae1f-db9478812213\") " Nov 02 18:25:15.565940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.565909 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca" pod "d5f437d6-10a2-4596-ae1f-db9478812213" (UID: "d5f437d6-10a2-4596-ae1f-db9478812213"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:15.566392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.566368 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5f437d6-10a2-4596-ae1f-db9478812213-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:15.568033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.567997 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs" pod "d5f437d6-10a2-4596-ae1f-db9478812213" (UID: "d5f437d6-10a2-4596-ae1f-db9478812213"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:25:15.568033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.568014 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls" pod "d5f437d6-10a2-4596-ae1f-db9478812213" (UID: "d5f437d6-10a2-4596-ae1f-db9478812213"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:25:15.568033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.568016 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns" pod "d5f437d6-10a2-4596-ae1f-db9478812213" (UID: "d5f437d6-10a2-4596-ae1f-db9478812213"). InnerVolumeSpecName "kube-api-access-2ldns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:15.667955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.667919 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:15.667955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.667949 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5f437d6-10a2-4596-ae1f-db9478812213-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:15.667955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.667959 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldns\" (UniqueName: \"kubernetes.io/projected/d5f437d6-10a2-4596-ae1f-db9478812213-kube-api-access-2ldns\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:15.822864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.822760 5647 generic.go:356] "Generic (PLEG): container finished" podID="d5f437d6-10a2-4596-ae1f-db9478812213" containerID="c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02" exitCode=0 Nov 02 18:25:15.822864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.822809 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" event={"ID":"d5f437d6-10a2-4596-ae1f-db9478812213","Type":"ContainerDied","Data":"c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02"} Nov 02 18:25:15.822864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.822831 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" Nov 02 18:25:15.822864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.822867 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql" event={"ID":"d5f437d6-10a2-4596-ae1f-db9478812213","Type":"ContainerDied","Data":"74ab18548763d4797f48186ee2f24e547eb88ddfea8a2d13eaaa269f9e2bf2f2"} Nov 02 18:25:15.823220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.822890 5647 scope.go:122] "RemoveContainer" containerID="c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02" Nov 02 18:25:15.834471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.834451 5647 scope.go:122] "RemoveContainer" containerID="c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02" Nov 02 18:25:15.834793 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:25:15.834773 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02\": container with ID starting with c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02 not found: ID does not exist" containerID="c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02" Nov 02 18:25:15.834863 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.834800 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02"} err="failed to get container status \"c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02\": rpc error: code = NotFound desc = could not find container \"c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02\": container with ID starting with c852d289749d510485a2f69c4b26473069a82772f35bea02d097b39f2d637d02 not found: ID does not exist" Nov 02 18:25:15.990063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.990033 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql"] Nov 02 18:25:15.994014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:15.993988 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-24a8b0-657c8bb798-mpbql"] Nov 02 18:25:17.081378 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:17.081339 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f437d6-10a2-4596-ae1f-db9478812213" path="/var/lib/kubelet/pods/d5f437d6-10a2-4596-ae1f-db9478812213/volumes" Nov 02 18:25:27.845758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:27.843171 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss"] Nov 02 18:25:27.845758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:27.843568 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" containerID="cri-o://668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800" gracePeriod=30 Nov 02 18:25:28.041847 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:28.041811 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:25:31.904940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:31.904903 5647 patch_prober.go:29] interesting pod/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:25:31.905769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:31.905710 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:25:33.145510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.145439 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:25:33.280128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280097 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280189 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280245 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280265 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280347 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280377 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280466 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280589 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:33.280668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280594 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280648 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs\") pod \"92354e2e-707d-4046-92d3-8468eca9b7eb\" (UID: \"92354e2e-707d-4046-92d3-8468eca9b7eb\") " Nov 02 18:25:33.280889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280507 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:25:33.280889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.280806 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:33.281018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281002 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:33.281148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281116 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:33.281448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281434 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.281525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281454 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.281525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281489 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.281525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281504 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.281525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.281518 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92354e2e-707d-4046-92d3-8468eca9b7eb-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.282947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.282897 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:25:33.283195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.283150 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:33.283305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.283207 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "kube-api-access-bxvwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:25:33.283384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.283359 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy" pod "92354e2e-707d-4046-92d3-8468eca9b7eb" (UID: "92354e2e-707d-4046-92d3-8468eca9b7eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:25:33.382661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.382628 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.382661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.382656 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvwr\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-kube-api-access-bxvwr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.382661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.382667 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.382939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.382675 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92354e2e-707d-4046-92d3-8468eca9b7eb-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:25:33.942289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.942248 5647 generic.go:356] "Generic (PLEG): container finished" podID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerID="668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800" exitCode=0 Nov 02 18:25:33.942507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.942328 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" Nov 02 18:25:33.942507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.942326 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" event={"ID":"92354e2e-707d-4046-92d3-8468eca9b7eb","Type":"ContainerDied","Data":"668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800"} Nov 02 18:25:33.942507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.942491 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss" event={"ID":"92354e2e-707d-4046-92d3-8468eca9b7eb","Type":"ContainerDied","Data":"a79b83f9e8504970255112129bb85c5c701ccc93ac1eaa8f8945e5e597a4b4a4"} Nov 02 18:25:33.942674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.942520 5647 scope.go:122] "RemoveContainer" containerID="668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800" Nov 02 18:25:33.954262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.954240 5647 scope.go:122] "RemoveContainer" containerID="87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe" Nov 02 18:25:33.964602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.964579 5647 scope.go:122] "RemoveContainer" containerID="668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800" Nov 02 18:25:33.964929 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:25:33.964907 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800\": container with ID starting with 668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800 not found: ID does not exist" containerID="668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800" Nov 02 18:25:33.965037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.964942 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800"} err="failed to get container status \"668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800\": rpc error: code = NotFound desc = could not find container \"668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800\": container with ID starting with 668ede8e744a551428649e2316b871bd96b2b63cfe146b359286aac3d7d6a800 not found: ID does not exist" Nov 02 18:25:33.965037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.964968 5647 scope.go:122] "RemoveContainer" containerID="87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe" Nov 02 18:25:33.965253 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:25:33.965230 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe\": container with ID starting with 87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe not found: ID does not exist" containerID="87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe" Nov 02 18:25:33.965298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:33.965263 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe"} err="failed to get container status \"87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe\": rpc error: code = NotFound desc = could not find container \"87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe\": container with ID starting with 87be81d1672e53522330be39c82afdfb4e7c7a31ddcf3eac72f7080688e5afbe not found: ID does not exist" Nov 02 18:25:34.111544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:34.111507 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss"] Nov 02 18:25:34.114062 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:34.114039 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-24a8b0-openshift-default-57c4b6ff69-96jss"] Nov 02 18:25:35.081941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:35.081904 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" path="/var/lib/kubelet/pods/92354e2e-707d-4046-92d3-8468eca9b7eb/volumes" Nov 02 18:25:35.949107 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:25:35.949067 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:25:36.660211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660176 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d84896d9-99kdc"] Nov 02 18:25:36.660860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660846 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d5f437d6-10a2-4596-ae1f-db9478812213" containerName="maas-api" Nov 02 18:25:36.660911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660861 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f437d6-10a2-4596-ae1f-db9478812213" containerName="maas-api" Nov 02 18:25:36.660911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660876 5647 cpu_manager.go:414] "containerMap: removing container" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" Nov 02 18:25:36.660911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660881 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" Nov 02 18:25:36.660911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660903 5647 cpu_manager.go:414] "containerMap: removing container" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" Nov 02 18:25:36.660911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.660908 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" Nov 02 18:25:36.661075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.661009 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f437d6-10a2-4596-ae1f-db9478812213" containerName="maas-api" Nov 02 18:25:36.661075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.661020 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" Nov 02 18:25:36.661075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.661029 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="92354e2e-707d-4046-92d3-8468eca9b7eb" containerName="istio-proxy" Nov 02 18:25:36.664583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.664566 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.719159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.719129 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ad0e4981cd4f\"" Nov 02 18:25:36.749467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.749430 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nqbm\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.749667 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.749626 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.749817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.749665 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.749817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.749749 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.769909 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.769870 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 18:25:36.769909 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.769897 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 18:25:36.770117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.769971 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 18:25:36.770117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.769994 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n74nw\"" Nov 02 18:25:36.770117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.769999 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 18:25:36.772982 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.772964 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d84896d9-99kdc"] Nov 02 18:25:36.851058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.851022 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.851273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.851066 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.851273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.851099 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.851407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.851271 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nqbm\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.851876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.851850 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.852000 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.851965 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.853702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.853681 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.909873 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.909842 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nqbm\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm\") pod \"payload-processing-external-model-6d84896d9-99kdc\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:36.979295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:36.979212 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:37.364489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:37.364456 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d84896d9-99kdc"] Nov 02 18:25:37.365372 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:25:37.365337 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613d6e03_c934_46e3_84ca_368e23f8506f.slice/crio-505cadd88b2604faab811cb8b975d6b7e11e47c25095fec62ee3b17d4c677a33 WatchSource:0}: Error finding container 505cadd88b2604faab811cb8b975d6b7e11e47c25095fec62ee3b17d4c677a33: Status 404 returned error can't find the container with id 505cadd88b2604faab811cb8b975d6b7e11e47c25095fec62ee3b17d4c677a33 Nov 02 18:25:37.972278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:37.972238 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" event={"ID":"613d6e03-c934-46e3-84ca-368e23f8506f","Type":"ContainerStarted","Data":"a6c8379bfa4fc90bc556bff4357ecb4c5b47a30ffdb1aeba58e71c9490ffd53d"} Nov 02 18:25:37.972714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:37.972306 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" event={"ID":"613d6e03-c934-46e3-84ca-368e23f8506f","Type":"ContainerStarted","Data":"505cadd88b2604faab811cb8b975d6b7e11e47c25095fec62ee3b17d4c677a33"} Nov 02 18:25:37.972714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:37.972334 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:25:38.096643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:38.096588 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" podStartSLOduration=2.096572894 podStartE2EDuration="2.096572894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:38.095137966 +0000 UTC m=+3725.168665170" watchObservedRunningTime="2026-11-02 18:25:38.096572894 +0000 UTC m=+3725.170100097" Nov 02 18:25:48.982004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:25:48.981966 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:26:05.947668 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:26:05.947622 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:26:09.180622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:09.180586 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:26:10.394683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.394649 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst"] Nov 02 18:26:10.398842 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.398817 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.488394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.488359 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst"] Nov 02 18:26:10.549159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549110 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549181 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549216 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549246 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglct\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549417 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549485 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549699 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549612 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549699 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549659 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.549889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.549700 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.651677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651563 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.651677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651622 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.651677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651664 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651847 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651909 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651940 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.651969 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglct\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.652054 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.652056 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.652124 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.652234 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652506 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.652478 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.652637 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.652625 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.653144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.653123 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.655096 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.655077 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.655245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.655216 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.710086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.710052 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.710251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.710179 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglct\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-g8qst\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:10.715920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:10.715893 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:11.263931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:11.263898 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst"] Nov 02 18:26:11.264073 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:26:11.263939 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod377b6b6a_3a4d_4b68_8b8b_bd40629ddf45.slice/crio-267047d2d74e165911df9a609931b021ff02e84a34f8d2cfe7ef174bec63ec76 WatchSource:0}: Error finding container 267047d2d74e165911df9a609931b021ff02e84a34f8d2cfe7ef174bec63ec76: Status 404 returned error can't find the container with id 267047d2d74e165911df9a609931b021ff02e84a34f8d2cfe7ef174bec63ec76 Nov 02 18:26:11.267049 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:11.267008 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:26:11.267127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:11.267100 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:26:11.267166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:11.267143 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:26:12.201210 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.201173 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" event={"ID":"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45","Type":"ContainerStarted","Data":"8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1"} Nov 02 18:26:12.201807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.201228 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" event={"ID":"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45","Type":"ContainerStarted","Data":"267047d2d74e165911df9a609931b021ff02e84a34f8d2cfe7ef174bec63ec76"} Nov 02 18:26:12.391288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.391231 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podStartSLOduration=2.391216622 podStartE2EDuration="2.391216622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:12.390172043 +0000 UTC m=+3759.463699247" watchObservedRunningTime="2026-11-02 18:26:12.391216622 +0000 UTC m=+3759.464743825" Nov 02 18:26:12.720797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.719147 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:12.748800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.740889 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:12.748800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.740964 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:12.971708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:12.971606 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d84896d9-99kdc"] Nov 02 18:26:13.724762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:13.724235 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:13.724762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:13.724300 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:14.722768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:14.722136 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:14.722768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:14.722213 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:15.726762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:15.723143 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:15.726762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:15.723206 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:16.720701 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:16.720663 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:16.721036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:16.721008 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:17.722576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:17.722113 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:17.722576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:17.722175 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:17.972968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:17.972873 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" podUID="613d6e03-c934-46e3-84ca-368e23f8506f" containerName="payload-processing" containerID="cri-o://a6c8379bfa4fc90bc556bff4357ecb4c5b47a30ffdb1aeba58e71c9490ffd53d" gracePeriod=25 Nov 02 18:26:18.261063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.261032 5647 generic.go:356] "Generic (PLEG): container finished" podID="613d6e03-c934-46e3-84ca-368e23f8506f" containerID="a6c8379bfa4fc90bc556bff4357ecb4c5b47a30ffdb1aeba58e71c9490ffd53d" exitCode=0 Nov 02 18:26:18.261212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.261098 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" event={"ID":"613d6e03-c934-46e3-84ca-368e23f8506f","Type":"ContainerDied","Data":"a6c8379bfa4fc90bc556bff4357ecb4c5b47a30ffdb1aeba58e71c9490ffd53d"} Nov 02 18:26:18.261212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.261146 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" event={"ID":"613d6e03-c934-46e3-84ca-368e23f8506f","Type":"ContainerDied","Data":"505cadd88b2604faab811cb8b975d6b7e11e47c25095fec62ee3b17d4c677a33"} Nov 02 18:26:18.261212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.261162 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505cadd88b2604faab811cb8b975d6b7e11e47c25095fec62ee3b17d4c677a33" Nov 02 18:26:18.263168 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.263143 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:26:18.309149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309119 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume\") pod \"613d6e03-c934-46e3-84ca-368e23f8506f\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " Nov 02 18:26:18.309316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309178 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm\") pod \"613d6e03-c934-46e3-84ca-368e23f8506f\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " Nov 02 18:26:18.309316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309285 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume\") pod \"613d6e03-c934-46e3-84ca-368e23f8506f\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " Nov 02 18:26:18.309316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309310 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials\") pod \"613d6e03-c934-46e3-84ca-368e23f8506f\" (UID: \"613d6e03-c934-46e3-84ca-368e23f8506f\") " Nov 02 18:26:18.309582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309541 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume" pod "613d6e03-c934-46e3-84ca-368e23f8506f" (UID: "613d6e03-c934-46e3-84ca-368e23f8506f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:18.309710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309660 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume" pod "613d6e03-c934-46e3-84ca-368e23f8506f" (UID: "613d6e03-c934-46e3-84ca-368e23f8506f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:18.309862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309840 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:18.309973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.309863 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/613d6e03-c934-46e3-84ca-368e23f8506f-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:18.311697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.311671 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials" pod "613d6e03-c934-46e3-84ca-368e23f8506f" (UID: "613d6e03-c934-46e3-84ca-368e23f8506f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:18.312062 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.312043 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm" pod "613d6e03-c934-46e3-84ca-368e23f8506f" (UID: "613d6e03-c934-46e3-84ca-368e23f8506f"). InnerVolumeSpecName "kube-api-access-9nqbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:18.410948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.410897 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nqbm\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-kube-api-access-9nqbm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:18.410948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.410942 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/613d6e03-c934-46e3-84ca-368e23f8506f-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:18.716522 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.716486 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-g8qst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 02 18:26:18.716702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:18.716549 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 02 18:26:19.267961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:19.267930 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d84896d9-99kdc" Nov 02 18:26:19.437622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:19.437514 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d84896d9-99kdc"] Nov 02 18:26:19.486511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:19.486468 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d84896d9-99kdc"] Nov 02 18:26:20.216282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:20.216250 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:20.218773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:20.217693 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:20.219086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:20.219053 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:26:21.081747 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:21.081701 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="613d6e03-c934-46e3-84ca-368e23f8506f" path="/var/lib/kubelet/pods/613d6e03-c934-46e3-84ca-368e23f8506f/volumes" Nov 02 18:26:35.950275 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:26:35.950235 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:26:40.589444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.589408 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk"] Nov 02 18:26:40.590138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.590115 5647 cpu_manager.go:414] "containerMap: removing container" podUID="613d6e03-c934-46e3-84ca-368e23f8506f" containerName="payload-processing" Nov 02 18:26:40.590138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.590136 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="613d6e03-c934-46e3-84ca-368e23f8506f" containerName="payload-processing" Nov 02 18:26:40.590338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.590244 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="613d6e03-c934-46e3-84ca-368e23f8506f" containerName="payload-processing" Nov 02 18:26:40.594023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.594001 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.649377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.649344 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 18:26:40.681199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.681163 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.681199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.681208 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvh9\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.681436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.681400 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.681608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.681584 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.701469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.701437 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0fc530495c8c\"" Nov 02 18:26:40.701723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.701446 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n74nw\"" Nov 02 18:26:40.701723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.701448 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4bef29666866\"" Nov 02 18:26:40.701723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.701454 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 18:26:40.701899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.701487 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 18:26:40.701899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.701525 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 18:26:40.704417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.704392 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk"] Nov 02 18:26:40.783190 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.783134 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.783421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.783340 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.783421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.783400 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.783551 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.783440 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvh9\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.783956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.783924 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.784101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.784084 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.786079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.786060 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.843602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.843525 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvh9\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9\") pod \"payload-processing-external-model-5f4b5cccd6-tsffk\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:40.910037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:40.909966 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:41.292785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:41.292749 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk"] Nov 02 18:26:41.294241 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:26:41.294202 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa44ab2f_3a78_46ce_9c0c_2f418b41b384.slice/crio-bc481c90c450d8f2a624630203a9e0558d8d716a4eab26b905cb85b1f160e891 WatchSource:0}: Error finding container bc481c90c450d8f2a624630203a9e0558d8d716a4eab26b905cb85b1f160e891: Status 404 returned error can't find the container with id bc481c90c450d8f2a624630203a9e0558d8d716a4eab26b905cb85b1f160e891 Nov 02 18:26:41.433186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:41.433152 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" event={"ID":"fa44ab2f-3a78-46ce-9c0c-2f418b41b384","Type":"ContainerStarted","Data":"749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed"} Nov 02 18:26:41.433381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:41.433201 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" event={"ID":"fa44ab2f-3a78-46ce-9c0c-2f418b41b384","Type":"ContainerStarted","Data":"bc481c90c450d8f2a624630203a9e0558d8d716a4eab26b905cb85b1f160e891"} Nov 02 18:26:41.433381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:41.433270 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:41.553078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:41.552973 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" podStartSLOduration=1.552952092 podStartE2EDuration="1.552952092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:41.552718829 +0000 UTC m=+3788.626246031" watchObservedRunningTime="2026-11-02 18:26:41.552952092 +0000 UTC m=+3788.626479325" Nov 02 18:26:45.743959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.743925 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk"] Nov 02 18:26:45.775572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.775542 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj"] Nov 02 18:26:45.780949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.780927 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.855417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.855383 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj"] Nov 02 18:26:45.873394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.873342 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.873598 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.873454 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.873598 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.873548 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.873944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.873914 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8lw\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.975184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.975138 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.975375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.975297 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8lw\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.975375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.975364 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.975474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.975433 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.976090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.976062 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.976206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.976063 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:45.978148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:45.978129 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:46.040092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:46.040021 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8lw\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw\") pod \"payload-processing-external-model-596fb6c5c4-rvmcj\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:46.096612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:46.096576 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:46.476284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:46.476252 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj"] Nov 02 18:26:46.477449 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:26:46.477425 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767b9707_2189_45fb_b90a_b6edc82cfea8.slice/crio-1c44cf1203447a7757a90f0aca8d1e130f3fa8a5c9e11a80c0b24ddc4cdfddc0 WatchSource:0}: Error finding container 1c44cf1203447a7757a90f0aca8d1e130f3fa8a5c9e11a80c0b24ddc4cdfddc0: Status 404 returned error can't find the container with id 1c44cf1203447a7757a90f0aca8d1e130f3fa8a5c9e11a80c0b24ddc4cdfddc0 Nov 02 18:26:47.475148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:47.475106 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" event={"ID":"767b9707-2189-45fb-b90a-b6edc82cfea8","Type":"ContainerStarted","Data":"f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5"} Nov 02 18:26:47.475610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:47.475185 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" event={"ID":"767b9707-2189-45fb-b90a-b6edc82cfea8","Type":"ContainerStarted","Data":"1c44cf1203447a7757a90f0aca8d1e130f3fa8a5c9e11a80c0b24ddc4cdfddc0"} Nov 02 18:26:47.475610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:47.475291 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:47.597218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:47.596788 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" podStartSLOduration=2.596774585 podStartE2EDuration="2.596774585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:47.5940491 +0000 UTC m=+3794.667576302" watchObservedRunningTime="2026-11-02 18:26:47.596774585 +0000 UTC m=+3794.670301788" Nov 02 18:26:50.744942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:50.744869 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" podUID="fa44ab2f-3a78-46ce-9c0c-2f418b41b384" containerName="payload-processing" containerID="cri-o://749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed" gracePeriod=25 Nov 02 18:26:51.011875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.011797 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:51.171650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.171608 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume\") pod \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " Nov 02 18:26:51.171950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.171671 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume\") pod \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " Nov 02 18:26:51.171950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.171726 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9\") pod \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " Nov 02 18:26:51.171950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.171804 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials\") pod \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\" (UID: \"fa44ab2f-3a78-46ce-9c0c-2f418b41b384\") " Nov 02 18:26:51.172187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.172076 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume" pod "fa44ab2f-3a78-46ce-9c0c-2f418b41b384" (UID: "fa44ab2f-3a78-46ce-9c0c-2f418b41b384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:51.172187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.172087 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume" pod "fa44ab2f-3a78-46ce-9c0c-2f418b41b384" (UID: "fa44ab2f-3a78-46ce-9c0c-2f418b41b384"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:51.172393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.172374 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:51.172450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.172395 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:51.174120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.174093 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9" pod "fa44ab2f-3a78-46ce-9c0c-2f418b41b384" (UID: "fa44ab2f-3a78-46ce-9c0c-2f418b41b384"). InnerVolumeSpecName "kube-api-access-psvh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:51.174213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.174132 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials" pod "fa44ab2f-3a78-46ce-9c0c-2f418b41b384" (UID: "fa44ab2f-3a78-46ce-9c0c-2f418b41b384"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:51.274313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.274229 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvh9\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-kube-api-access-psvh9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:51.274313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.274267 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa44ab2f-3a78-46ce-9c0c-2f418b41b384-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:51.545508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.545424 5647 generic.go:356] "Generic (PLEG): container finished" podID="fa44ab2f-3a78-46ce-9c0c-2f418b41b384" containerID="749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed" exitCode=0 Nov 02 18:26:51.545508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.545469 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" Nov 02 18:26:51.545714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.545469 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" event={"ID":"fa44ab2f-3a78-46ce-9c0c-2f418b41b384","Type":"ContainerDied","Data":"749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed"} Nov 02 18:26:51.545714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.545603 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk" event={"ID":"fa44ab2f-3a78-46ce-9c0c-2f418b41b384","Type":"ContainerDied","Data":"bc481c90c450d8f2a624630203a9e0558d8d716a4eab26b905cb85b1f160e891"} Nov 02 18:26:51.545714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.545633 5647 scope.go:122] "RemoveContainer" containerID="749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed" Nov 02 18:26:51.556915 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.556892 5647 scope.go:122] "RemoveContainer" containerID="749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed" Nov 02 18:26:51.557223 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:26:51.557202 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed\": container with ID starting with 749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed not found: ID does not exist" containerID="749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed" Nov 02 18:26:51.557287 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.557232 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed"} err="failed to get container status \"749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed\": rpc error: code = NotFound desc = could not find container \"749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed\": container with ID starting with 749947a00440a269c5ca2b57f0f91982fea5629abb8a948ca6636f4c67c1b2ed not found: ID does not exist" Nov 02 18:26:51.716850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.716803 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk"] Nov 02 18:26:51.719503 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:51.719475 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5f4b5cccd6-tsffk"] Nov 02 18:26:53.081457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:53.081426 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa44ab2f-3a78-46ce-9c0c-2f418b41b384" path="/var/lib/kubelet/pods/fa44ab2f-3a78-46ce-9c0c-2f418b41b384/volumes" Nov 02 18:26:53.538869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:53.538831 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj"] Nov 02 18:26:53.541984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:53.541950 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" podUID="767b9707-2189-45fb-b90a-b6edc82cfea8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:26:58.540024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.539958 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" podUID="767b9707-2189-45fb-b90a-b6edc82cfea8" containerName="payload-processing" containerID="cri-o://f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5" gracePeriod=25 Nov 02 18:26:58.865356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.865324 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:58.975715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.975670 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume\") pod \"767b9707-2189-45fb-b90a-b6edc82cfea8\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " Nov 02 18:26:58.975975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.975950 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw\") pod \"767b9707-2189-45fb-b90a-b6edc82cfea8\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " Nov 02 18:26:58.976076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.976060 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume\") pod \"767b9707-2189-45fb-b90a-b6edc82cfea8\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " Nov 02 18:26:58.976168 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.976153 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials\") pod \"767b9707-2189-45fb-b90a-b6edc82cfea8\" (UID: \"767b9707-2189-45fb-b90a-b6edc82cfea8\") " Nov 02 18:26:58.976168 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.976158 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume" pod "767b9707-2189-45fb-b90a-b6edc82cfea8" (UID: "767b9707-2189-45fb-b90a-b6edc82cfea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:58.976448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.976407 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume" pod "767b9707-2189-45fb-b90a-b6edc82cfea8" (UID: "767b9707-2189-45fb-b90a-b6edc82cfea8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:26:58.976965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.976940 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:58.976965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.976968 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767b9707-2189-45fb-b90a-b6edc82cfea8-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:58.978450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.978426 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials" pod "767b9707-2189-45fb-b90a-b6edc82cfea8" (UID: "767b9707-2189-45fb-b90a-b6edc82cfea8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:58.978572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:58.978470 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw" pod "767b9707-2189-45fb-b90a-b6edc82cfea8" (UID: "767b9707-2189-45fb-b90a-b6edc82cfea8"). InnerVolumeSpecName "kube-api-access-8p8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:26:59.078085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.077991 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:59.078085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.078024 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p8lw\" (UniqueName: \"kubernetes.io/projected/767b9707-2189-45fb-b90a-b6edc82cfea8-kube-api-access-8p8lw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:26:59.605693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.605640 5647 generic.go:356] "Generic (PLEG): container finished" podID="767b9707-2189-45fb-b90a-b6edc82cfea8" containerID="f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5" exitCode=0 Nov 02 18:26:59.606247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.605843 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" event={"ID":"767b9707-2189-45fb-b90a-b6edc82cfea8","Type":"ContainerDied","Data":"f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5"} Nov 02 18:26:59.606247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.605899 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" event={"ID":"767b9707-2189-45fb-b90a-b6edc82cfea8","Type":"ContainerDied","Data":"1c44cf1203447a7757a90f0aca8d1e130f3fa8a5c9e11a80c0b24ddc4cdfddc0"} Nov 02 18:26:59.606247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.605934 5647 scope.go:122] "RemoveContainer" containerID="f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5" Nov 02 18:26:59.606247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.606107 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj" Nov 02 18:26:59.630166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.630137 5647 scope.go:122] "RemoveContainer" containerID="f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5" Nov 02 18:26:59.631138 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:26:59.631111 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5\": container with ID starting with f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5 not found: ID does not exist" containerID="f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5" Nov 02 18:26:59.631335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.631304 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5"} err="failed to get container status \"f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5\": rpc error: code = NotFound desc = could not find container \"f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5\": container with ID starting with f5c7eb02906539b86c97a6107f05f780e65bce0eafd91607b0ec8318ec37afd5 not found: ID does not exist" Nov 02 18:26:59.775791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.775758 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj"] Nov 02 18:26:59.782102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:26:59.782069 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-596fb6c5c4-rvmcj"] Nov 02 18:27:01.081604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.081567 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="767b9707-2189-45fb-b90a-b6edc82cfea8" path="/var/lib/kubelet/pods/767b9707-2189-45fb-b90a-b6edc82cfea8/volumes" Nov 02 18:27:01.359153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359125 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw"] Nov 02 18:27:01.359764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359751 5647 cpu_manager.go:414] "containerMap: removing container" podUID="767b9707-2189-45fb-b90a-b6edc82cfea8" containerName="payload-processing" Nov 02 18:27:01.359823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359765 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="767b9707-2189-45fb-b90a-b6edc82cfea8" containerName="payload-processing" Nov 02 18:27:01.359823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359788 5647 cpu_manager.go:414] "containerMap: removing container" podUID="fa44ab2f-3a78-46ce-9c0c-2f418b41b384" containerName="payload-processing" Nov 02 18:27:01.359823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359793 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa44ab2f-3a78-46ce-9c0c-2f418b41b384" containerName="payload-processing" Nov 02 18:27:01.359936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359896 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="767b9707-2189-45fb-b90a-b6edc82cfea8" containerName="payload-processing" Nov 02 18:27:01.359936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.359907 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa44ab2f-3a78-46ce-9c0c-2f418b41b384" containerName="payload-processing" Nov 02 18:27:01.365178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.365142 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.418813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.418785 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4b8a8a727efd\"" Nov 02 18:27:01.442708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.442662 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.442978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.442894 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.443039 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.443002 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv77\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.443093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.443076 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.469480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.469392 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 02 18:27:01.469480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.469409 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n74nw\"" Nov 02 18:27:01.469480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.469409 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 02 18:27:01.469480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.469437 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4bef29666866\"" Nov 02 18:27:01.469480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.469450 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 02 18:27:01.470099 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.470064 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 02 18:27:01.474418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.474379 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw"] Nov 02 18:27:01.544264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.544228 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.544452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.544332 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.544452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.544368 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv77\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.544452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.544402 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.545131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.545053 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.545306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.545182 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.546968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.546947 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.604178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.604146 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv77\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77\") pod \"payload-processing-external-model-55cbdcdf67-gv4kw\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:01.681369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:01.681295 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:02.060792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:02.060759 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw"] Nov 02 18:27:02.061906 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:27:02.061879 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926c0c80_d76e_42e0_9828_048136075ac5.slice/crio-e3417793895b79a77641b8edcc512951a9d2b2a09458c62a52e79097b3cddd6e WatchSource:0}: Error finding container e3417793895b79a77641b8edcc512951a9d2b2a09458c62a52e79097b3cddd6e: Status 404 returned error can't find the container with id e3417793895b79a77641b8edcc512951a9d2b2a09458c62a52e79097b3cddd6e Nov 02 18:27:02.629976 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:02.629932 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" event={"ID":"926c0c80-d76e-42e0-9828-048136075ac5","Type":"ContainerStarted","Data":"49faac55185efa981844f7828038edf90646c73a797167b1707220f3e164f1d7"} Nov 02 18:27:02.630421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:02.629991 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" event={"ID":"926c0c80-d76e-42e0-9828-048136075ac5","Type":"ContainerStarted","Data":"e3417793895b79a77641b8edcc512951a9d2b2a09458c62a52e79097b3cddd6e"} Nov 02 18:27:02.630421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:02.630143 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:02.751776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:02.751701 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" podStartSLOduration=1.751685489 podStartE2EDuration="1.751685489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:02.751497536 +0000 UTC m=+3809.825024740" watchObservedRunningTime="2026-11-02 18:27:02.751685489 +0000 UTC m=+3809.825212691" Nov 02 18:27:06.011307 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:06.010484 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:27:06.516652 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:06.516614 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw"] Nov 02 18:27:10.767147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:10.767110 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524"] Nov 02 18:27:10.767630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:10.767334 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" podUID="01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" containerName="maas-api" containerID="cri-o://7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8" gracePeriod=30 Nov 02 18:27:11.103866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.103832 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:27:11.264276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.264228 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs\") pod \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " Nov 02 18:27:11.264483 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.264432 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca\") pod \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " Nov 02 18:27:11.264550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.264521 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls\") pod \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " Nov 02 18:27:11.264624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.264605 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh\" (UniqueName: \"kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh\") pod \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\" (UID: \"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040\") " Nov 02 18:27:11.298359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.298217 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca" pod "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" (UID: "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:11.298794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.298723 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls" pod "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" (UID: "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:27:11.298960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.298877 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs" pod "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" (UID: "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:27:11.299030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.298974 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh" pod "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" (UID: "01ee3dfb-5cfc-45fc-9ad1-6a41d809e040"). InnerVolumeSpecName "kube-api-access-gpbwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:11.369763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.369201 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:11.369763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.369237 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:11.369763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.369252 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:11.369763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.369266 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpbwh\" (UniqueName: \"kubernetes.io/projected/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040-kube-api-access-gpbwh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:11.517313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.517271 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" podUID="926c0c80-d76e-42e0-9828-048136075ac5" containerName="payload-processing" containerID="cri-o://49faac55185efa981844f7828038edf90646c73a797167b1707220f3e164f1d7" gracePeriod=25 Nov 02 18:27:11.724691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.724652 5647 generic.go:356] "Generic (PLEG): container finished" podID="01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" containerID="7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8" exitCode=0 Nov 02 18:27:11.724981 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.724959 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" Nov 02 18:27:11.726762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.725913 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" event={"ID":"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040","Type":"ContainerDied","Data":"7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8"} Nov 02 18:27:11.726762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.725994 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524" event={"ID":"01ee3dfb-5cfc-45fc-9ad1-6a41d809e040","Type":"ContainerDied","Data":"b501bb597fd89b4715223c28149426c7c9e9207e83ba0a9bac0a0fec7bea4564"} Nov 02 18:27:11.726762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.726032 5647 scope.go:122] "RemoveContainer" containerID="7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8" Nov 02 18:27:11.728722 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.728692 5647 generic.go:356] "Generic (PLEG): container finished" podID="926c0c80-d76e-42e0-9828-048136075ac5" containerID="49faac55185efa981844f7828038edf90646c73a797167b1707220f3e164f1d7" exitCode=0 Nov 02 18:27:11.728883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.728808 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" event={"ID":"926c0c80-d76e-42e0-9828-048136075ac5","Type":"ContainerDied","Data":"49faac55185efa981844f7828038edf90646c73a797167b1707220f3e164f1d7"} Nov 02 18:27:11.740518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.740495 5647 scope.go:122] "RemoveContainer" containerID="7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8" Nov 02 18:27:11.741094 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:11.741072 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8\": container with ID starting with 7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8 not found: ID does not exist" containerID="7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8" Nov 02 18:27:11.741189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.741100 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8"} err="failed to get container status \"7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8\": rpc error: code = NotFound desc = could not find container \"7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8\": container with ID starting with 7c3f7bf182032f748eb190440ed480816b1131b6a7a5e3ec1fbde6a5e24676e8 not found: ID does not exist" Nov 02 18:27:11.798988 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.798962 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:11.876063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876032 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume\") pod \"926c0c80-d76e-42e0-9828-048136075ac5\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " Nov 02 18:27:11.876261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876133 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77\") pod \"926c0c80-d76e-42e0-9828-048136075ac5\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " Nov 02 18:27:11.876261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876233 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials\") pod \"926c0c80-d76e-42e0-9828-048136075ac5\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " Nov 02 18:27:11.876368 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876269 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume\") pod \"926c0c80-d76e-42e0-9828-048136075ac5\" (UID: \"926c0c80-d76e-42e0-9828-048136075ac5\") " Nov 02 18:27:11.876431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876411 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume" pod "926c0c80-d76e-42e0-9828-048136075ac5" (UID: "926c0c80-d76e-42e0-9828-048136075ac5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:11.876891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876862 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume" pod "926c0c80-d76e-42e0-9828-048136075ac5" (UID: "926c0c80-d76e-42e0-9828-048136075ac5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:11.877612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.876925 5647 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-routing-overlay-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:11.878607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.878585 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials" pod "926c0c80-d76e-42e0-9828-048136075ac5" (UID: "926c0c80-d76e-42e0-9828-048136075ac5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:11.878937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.878919 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77" pod "926c0c80-d76e-42e0-9828-048136075ac5" (UID: "926c0c80-d76e-42e0-9828-048136075ac5"). InnerVolumeSpecName "kube-api-access-5vv77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:11.893510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.893485 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524"] Nov 02 18:27:11.955896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:11.952646 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a8da6e-697b544656-7w524"] Nov 02 18:27:12.023758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.006006 5647 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-provider-credentials\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:12.023758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.006046 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/926c0c80-d76e-42e0-9828-048136075ac5-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:12.023758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.006065 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vv77\" (UniqueName: \"kubernetes.io/projected/926c0c80-d76e-42e0-9828-048136075ac5-kube-api-access-5vv77\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:12.737957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.737927 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" Nov 02 18:27:12.737957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.737949 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw" event={"ID":"926c0c80-d76e-42e0-9828-048136075ac5","Type":"ContainerDied","Data":"e3417793895b79a77641b8edcc512951a9d2b2a09458c62a52e79097b3cddd6e"} Nov 02 18:27:12.738191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.738004 5647 scope.go:122] "RemoveContainer" containerID="49faac55185efa981844f7828038edf90646c73a797167b1707220f3e164f1d7" Nov 02 18:27:12.908680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.908641 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw"] Nov 02 18:27:12.911345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:12.911321 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55cbdcdf67-gv4kw"] Nov 02 18:27:13.082664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:13.082571 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" path="/var/lib/kubelet/pods/01ee3dfb-5cfc-45fc-9ad1-6a41d809e040/volumes" Nov 02 18:27:13.083108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:13.083086 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926c0c80-d76e-42e0-9828-048136075ac5" path="/var/lib/kubelet/pods/926c0c80-d76e-42e0-9828-048136075ac5/volumes" Nov 02 18:27:13.203118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:13.203085 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg"] Nov 02 18:27:13.203614 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:13.203581 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" containerID="cri-o://231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d" gracePeriod=30 Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.161509 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq"] Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.164014 5647 cpu_manager.go:414] "containerMap: removing container" podUID="926c0c80-d76e-42e0-9828-048136075ac5" containerName="payload-processing" Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.164039 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926c0c80-d76e-42e0-9828-048136075ac5" containerName="payload-processing" Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.164080 5647 cpu_manager.go:414] "containerMap: removing container" podUID="01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" containerName="maas-api" Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.164088 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" containerName="maas-api" Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.164305 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="926c0c80-d76e-42e0-9828-048136075ac5" containerName="payload-processing" Nov 02 18:27:17.165755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.164326 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ee3dfb-5cfc-45fc-9ad1-6a41d809e040" containerName="maas-api" Nov 02 18:27:17.189759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.188182 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.226589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.219245 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq"] Nov 02 18:27:17.230124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.229598 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg6v2\" (UniqueName: \"kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2\") pod \"payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.230124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.229916 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.248711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.244135 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-662a80\"" Nov 02 18:27:17.248711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.244978 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-662a80-dockercfg-7xgdx\"" Nov 02 18:27:17.330708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.330673 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg6v2\" (UniqueName: \"kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2\") pod \"payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.330911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.330854 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.331517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.331499 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.390487 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.390455 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg6v2\" (UniqueName: \"kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2\") pod \"payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.551721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.551614 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:17.940624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:17.940597 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq"] Nov 02 18:27:17.941658 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:27:17.941628 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515f836f_91bf_45aa_a4a9_e8eaed8a1b0d.slice/crio-c88988b771a27ad123a7d4214bff8853a52ced446858c43f911fbb12c6de62b8 WatchSource:0}: Error finding container c88988b771a27ad123a7d4214bff8853a52ced446858c43f911fbb12c6de62b8: Status 404 returned error can't find the container with id c88988b771a27ad123a7d4214bff8853a52ced446858c43f911fbb12c6de62b8 Nov 02 18:27:18.485314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.485242 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:27:18.567990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.567690 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.567990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.567767 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.567990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.567831 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.567990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.567878 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.567990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.567936 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.568418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.568008 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.568418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.568272 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.568418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.568334 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.568418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.568390 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket\") pod \"f9166951-3f12-47ff-b3a9-75c1d616f309\" (UID: \"f9166951-3f12-47ff-b3a9-75c1d616f309\") " Nov 02 18:27:18.569585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.569552 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:18.571871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.569968 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:18.571871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.570156 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:18.571871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.570623 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:18.571871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.571301 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:18.598046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.594750 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:27:18.598046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.597776 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:18.598046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.597947 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:18.598046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.598010 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7" pod "f9166951-3f12-47ff-b3a9-75c1d616f309" (UID: "f9166951-3f12-47ff-b3a9-75c1d616f309"). InnerVolumeSpecName "kube-api-access-ztzs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:18.671294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671250 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671294 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztzs7\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-kube-api-access-ztzs7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671310 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671325 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9166951-3f12-47ff-b3a9-75c1d616f309-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671338 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671350 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671362 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671375 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.671519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.671389 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9166951-3f12-47ff-b3a9-75c1d616f309-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:18.816850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.816765 5647 generic.go:356] "Generic (PLEG): container finished" podID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerID="231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d" exitCode=0 Nov 02 18:27:18.816850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.816811 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" Nov 02 18:27:18.817067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.816843 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" event={"ID":"f9166951-3f12-47ff-b3a9-75c1d616f309","Type":"ContainerDied","Data":"231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d"} Nov 02 18:27:18.817067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.816918 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg" event={"ID":"f9166951-3f12-47ff-b3a9-75c1d616f309","Type":"ContainerDied","Data":"586207df4d7c4f2a4ce08752c0fd6d4c22067e860aee0ecc23119a09456e3cef"} Nov 02 18:27:18.817067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.816945 5647 scope.go:122] "RemoveContainer" containerID="231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d" Nov 02 18:27:18.818936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.818906 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" event={"ID":"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d","Type":"ContainerStarted","Data":"14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5"} Nov 02 18:27:18.819064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.818960 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" event={"ID":"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d","Type":"ContainerStarted","Data":"c88988b771a27ad123a7d4214bff8853a52ced446858c43f911fbb12c6de62b8"} Nov 02 18:27:18.819064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.819009 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:18.828284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.828262 5647 scope.go:122] "RemoveContainer" containerID="244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329" Nov 02 18:27:18.838069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.838048 5647 scope.go:122] "RemoveContainer" containerID="231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d" Nov 02 18:27:18.838334 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:18.838306 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d\": container with ID starting with 231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d not found: ID does not exist" containerID="231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d" Nov 02 18:27:18.838431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.838347 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d"} err="failed to get container status \"231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d\": rpc error: code = NotFound desc = could not find container \"231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d\": container with ID starting with 231222eb9f0c319ae0259c4f9e594fbced505a12d751eaa79f8bdf435655a22d not found: ID does not exist" Nov 02 18:27:18.838431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.838375 5647 scope.go:122] "RemoveContainer" containerID="244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329" Nov 02 18:27:18.838601 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:18.838584 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329\": container with ID starting with 244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329 not found: ID does not exist" containerID="244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329" Nov 02 18:27:18.838641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.838607 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329"} err="failed to get container status \"244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329\": rpc error: code = NotFound desc = could not find container \"244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329\": container with ID starting with 244931c3745766557a823c38f10c5c64a9c2bc09d7e5de10f59ac295fa592329 not found: ID does not exist" Nov 02 18:27:18.987213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.987175 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg"] Nov 02 18:27:18.991045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:18.991015 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a8da6e-openshift-default-7cb99d5b77-bhfpg"] Nov 02 18:27:19.082353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:19.082272 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" path="/var/lib/kubelet/pods/f9166951-3f12-47ff-b3a9-75c1d616f309/volumes" Nov 02 18:27:19.105191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:19.105139 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" podStartSLOduration=2.105125071 podStartE2EDuration="2.105125071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:19.104174998 +0000 UTC m=+3826.177702228" watchObservedRunningTime="2026-11-02 18:27:19.105125071 +0000 UTC m=+3826.178652273" Nov 02 18:27:25.975409 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:25.975375 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg"] Nov 02 18:27:25.976080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:25.976064 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" Nov 02 18:27:25.976132 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:25.976084 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" Nov 02 18:27:25.976235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:25.976224 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" Nov 02 18:27:25.976274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:25.976244 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" Nov 02 18:27:25.981205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:25.981187 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087162 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087302 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087329 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087378 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmjk\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087513 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087637 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087677 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087709 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.087859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.087810 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189266 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189228 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189266 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189277 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189307 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmjk\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189404 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189540 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189591 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.189649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189630 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189673 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189713 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189847 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.189968 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.190551 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.190666 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.190883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.190811 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.198098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.197963 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.199438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.199101 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg"] Nov 02 18:27:26.207391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.207364 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.260687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.260621 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.261005 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.260983 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmjk\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-jwkvg\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.296707 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.296673 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:26.732558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.732519 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg"] Nov 02 18:27:26.734087 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:27:26.734051 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ef0310_95fb_48c0_82d0_8008c5131787.slice/crio-c09b8d007b4762eb7f09000c3c79d79ff7ebb640875b900394b74a0055717c09 WatchSource:0}: Error finding container c09b8d007b4762eb7f09000c3c79d79ff7ebb640875b900394b74a0055717c09: Status 404 returned error can't find the container with id c09b8d007b4762eb7f09000c3c79d79ff7ebb640875b900394b74a0055717c09 Nov 02 18:27:26.736841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.736804 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:26.736932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.736878 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:26.736932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.736907 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:26.872547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.872510 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" event={"ID":"b0ef0310-95fb-48c0-82d0-8008c5131787","Type":"ContainerStarted","Data":"ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa"} Nov 02 18:27:26.872758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:26.872571 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" event={"ID":"b0ef0310-95fb-48c0-82d0-8008c5131787","Type":"ContainerStarted","Data":"c09b8d007b4762eb7f09000c3c79d79ff7ebb640875b900394b74a0055717c09"} Nov 02 18:27:27.297337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:27.297296 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:27.298917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:27.298891 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:27.299079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:27.298945 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:28.298457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.298046 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:28.298457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.298103 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:28.336139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.336063 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podStartSLOduration=3.336039302 podStartE2EDuration="3.336039302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:26.999649471 +0000 UTC m=+3834.073176675" watchObservedRunningTime="2026-11-02 18:27:28.336039302 +0000 UTC m=+3835.409566506" Nov 02 18:27:28.336436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.336416 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf"] Nov 02 18:27:28.336649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.336625 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" containerID="cri-o://10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435" gracePeriod=30 Nov 02 18:27:28.340632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.340593 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d"] Nov 02 18:27:28.340913 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:28.340886 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" containerID="cri-o://b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f" gracePeriod=30 Nov 02 18:27:29.297978 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.297942 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:29.298170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.297997 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:29.731759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.730908 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7"] Nov 02 18:27:29.732308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.731917 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" Nov 02 18:27:29.732308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.731940 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9166951-3f12-47ff-b3a9-75c1d616f309" containerName="istio-proxy" Nov 02 18:27:29.736406 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.736368 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.781498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781459 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcvw\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.781726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781521 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.781726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781590 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.781726 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781633 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.781945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781771 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.782007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781935 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.782007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.781993 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.782204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.782078 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.782204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.782106 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.791855 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.791826 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-546881-openshift-default-dockercfg-rwhvb\"" Nov 02 18:27:29.794334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.794307 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7"] Nov 02 18:27:29.836065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.836022 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.906868 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcvw\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.906942 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907013 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907062 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907105 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907311 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907360 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907450 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.907479 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.908066 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.909079 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.911588 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.912826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.911896 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.950598 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.950520 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.959476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.959376 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:29.997832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:29.992014 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:30.049761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.048481 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcvw\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:30.049761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.048563 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token\") pod \"e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:30.061761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.061093 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:30.301055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.298938 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:30.301055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.299001 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:30.635002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.634970 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7"] Nov 02 18:27:30.635655 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:27:30.635628 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafb611c_d65f_447d_b126_881d5e3749f4.slice/crio-766c0bd64172192723b30221e4e235f1358fbed9bd96bcdda4c9d17200f99b86 WatchSource:0}: Error finding container 766c0bd64172192723b30221e4e235f1358fbed9bd96bcdda4c9d17200f99b86: Status 404 returned error can't find the container with id 766c0bd64172192723b30221e4e235f1358fbed9bd96bcdda4c9d17200f99b86 Nov 02 18:27:30.638558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.638524 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:30.638632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.638613 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:30.638672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.638644 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:30.994092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.994005 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" event={"ID":"aafb611c-d65f-447d-b126-881d5e3749f4","Type":"ContainerStarted","Data":"f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e"} Nov 02 18:27:30.994092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:30.994075 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" event={"ID":"aafb611c-d65f-447d-b126-881d5e3749f4","Type":"ContainerStarted","Data":"766c0bd64172192723b30221e4e235f1358fbed9bd96bcdda4c9d17200f99b86"} Nov 02 18:27:31.062557 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:31.062522 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:31.064226 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:31.064195 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 02 18:27:31.064379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:31.064241 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 02 18:27:31.131126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:31.131055 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podStartSLOduration=2.131036323 podStartE2EDuration="2.131036323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:31.129346396 +0000 UTC m=+3838.202873624" watchObservedRunningTime="2026-11-02 18:27:31.131036323 +0000 UTC m=+3838.204563526" Nov 02 18:27:31.297135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:31.297032 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:31.297135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:31.297102 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:32.067705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:32.067672 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:32.297018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:32.296981 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:32.297226 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:32.297053 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:33.007695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.007656 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:33.008695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.008673 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:27:33.297245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.297161 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:33.297245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.297207 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:33.667121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.667091 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:27:33.687088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.687057 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:27:33.780273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780227 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780320 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780357 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780387 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780424 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780467 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780497 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780659 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780675 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780707 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780781 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780793 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780833 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780865 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780904 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780906 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.780967 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781038 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781075 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket\") pod \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\" (UID: \"cb0ce187-e98e-4042-b957-a6f1f53b92d8\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781117 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.781156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781158 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert\") pod \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\" (UID: \"6c9ae0d2-d57a-4afd-b54a-6344f64b5004\") " Nov 02 18:27:33.782437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781932 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.782437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781955 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.782437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781971 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.782437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781985 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.784191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781076 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.784191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.781288 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:33.784396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.782286 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:33.784396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.782892 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.784396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.783584 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.784396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.783947 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.784396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.784150 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.784702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.784406 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:27:33.785771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.785705 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:33.785879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.785774 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:27:33.785879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.785773 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:27:33.786337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.786159 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:33.787779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.787141 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg" pod "cb0ce187-e98e-4042-b957-a6f1f53b92d8" (UID: "cb0ce187-e98e-4042-b957-a6f1f53b92d8"). InnerVolumeSpecName "kube-api-access-cn8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:33.789128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.789080 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8" pod "6c9ae0d2-d57a-4afd-b54a-6344f64b5004" (UID: "6c9ae0d2-d57a-4afd-b54a-6344f64b5004"). InnerVolumeSpecName "kube-api-access-7z8q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:33.883561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883525 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883560 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883574 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883609 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn8jg\" (UniqueName: \"kubernetes.io/projected/cb0ce187-e98e-4042-b957-a6f1f53b92d8-kube-api-access-cn8jg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883623 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883635 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883647 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883659 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883671 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883682 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb0ce187-e98e-4042-b957-a6f1f53b92d8-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883695 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883707 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb0ce187-e98e-4042-b957-a6f1f53b92d8-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883720 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z8q8\" (UniqueName: \"kubernetes.io/projected/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-kube-api-access-7z8q8\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:33.883858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:33.883752 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c9ae0d2-d57a-4afd-b54a-6344f64b5004-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:34.017178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.017137 5647 generic.go:356] "Generic (PLEG): container finished" podID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerID="b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f" exitCode=0 Nov 02 18:27:34.017393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.017219 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" Nov 02 18:27:34.017393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.017273 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" event={"ID":"6c9ae0d2-d57a-4afd-b54a-6344f64b5004","Type":"ContainerDied","Data":"b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f"} Nov 02 18:27:34.017516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.017366 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d" event={"ID":"6c9ae0d2-d57a-4afd-b54a-6344f64b5004","Type":"ContainerDied","Data":"c27a559941c3e1ba1237735b7aa232244e9dd8ebf068a63ff1fee09285db03d5"} Nov 02 18:27:34.017516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.017438 5647 scope.go:122] "RemoveContainer" containerID="b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f" Nov 02 18:27:34.019343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.019316 5647 generic.go:356] "Generic (PLEG): container finished" podID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerID="10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435" exitCode=0 Nov 02 18:27:34.019930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.019854 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" Nov 02 18:27:34.019930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.019915 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" event={"ID":"cb0ce187-e98e-4042-b957-a6f1f53b92d8","Type":"ContainerDied","Data":"10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435"} Nov 02 18:27:34.020134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.019970 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf" event={"ID":"cb0ce187-e98e-4042-b957-a6f1f53b92d8","Type":"ContainerDied","Data":"0369ddb31cf08e33be93ef71ffacfece3cdd2f1c122975ec7d1c89659ba9927e"} Nov 02 18:27:34.030557 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.030382 5647 scope.go:122] "RemoveContainer" containerID="b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f" Nov 02 18:27:34.030864 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:34.030832 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f\": container with ID starting with b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f not found: ID does not exist" containerID="b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f" Nov 02 18:27:34.031067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.030875 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f"} err="failed to get container status \"b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f\": rpc error: code = NotFound desc = could not find container \"b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f\": container with ID starting with b441ead6eb1afe9a27d64a07a3602e3ce4f0a805390f6530bb946d623ceb951f not found: ID does not exist" Nov 02 18:27:34.031067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.030901 5647 scope.go:122] "RemoveContainer" containerID="10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435" Nov 02 18:27:34.051030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.047014 5647 scope.go:122] "RemoveContainer" containerID="10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435" Nov 02 18:27:34.051030 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:34.047383 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435\": container with ID starting with 10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435 not found: ID does not exist" containerID="10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435" Nov 02 18:27:34.051030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.047422 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435"} err="failed to get container status \"10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435\": rpc error: code = NotFound desc = could not find container \"10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435\": container with ID starting with 10f5553863b7d39d24df39babf43b28c38286d59a793007cd19b863cf555c435 not found: ID does not exist" Nov 02 18:27:34.198172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.198053 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d"] Nov 02 18:27:34.199738 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.199709 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-mgt8d"] Nov 02 18:27:34.297879 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.297845 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body= Nov 02 18:27:34.298286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.297912 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" Nov 02 18:27:34.312282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.312251 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf"] Nov 02 18:27:34.314908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:34.314881 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e4b90c-openshift-default-57f75f694f-49xmf"] Nov 02 18:27:35.089810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:35.089771 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" path="/var/lib/kubelet/pods/6c9ae0d2-d57a-4afd-b54a-6344f64b5004/volumes" Nov 02 18:27:35.090397 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:35.090377 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" path="/var/lib/kubelet/pods/cb0ce187-e98e-4042-b957-a6f1f53b92d8/volumes" Nov 02 18:27:35.808279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:35.808247 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:35.809033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:35.809002 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:35.810219 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:35.810191 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:27:35.956586 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:27:35.956546 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:27:41.097100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097068 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96"] Nov 02 18:27:41.097694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097678 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" Nov 02 18:27:41.097694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097693 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" Nov 02 18:27:41.097898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097714 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" Nov 02 18:27:41.097898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097721 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" Nov 02 18:27:41.097898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097832 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0ce187-e98e-4042-b957-a6f1f53b92d8" containerName="istio-proxy" Nov 02 18:27:41.097898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.097842 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9ae0d2-d57a-4afd-b54a-6344f64b5004" containerName="istio-proxy" Nov 02 18:27:41.102607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.102584 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.195025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.194993 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96"] Nov 02 18:27:41.258198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258157 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258199 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258360 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258416 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258480 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258635 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258622 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258645 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgpw\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258687 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.258870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.258857 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.359850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.359725 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.359850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.359817 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.359850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.359838 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.359950 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.359980 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgpw\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360041 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360123 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360179 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360198 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360224 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360275 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360463 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.360607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.360532 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.361048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.361026 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.362436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.362406 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.362602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.362585 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.418526 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.418494 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgpw\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.418526 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.418528 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-s4p96\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:41.716695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:41.716602 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:42.156684 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:42.156649 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96"] Nov 02 18:27:42.157328 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:27:42.157297 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f84129a_bb35_4c76_9db8_bb15ea5fcd36.slice/crio-7ad8aa3e3ab54f2971b49fe26be5d90ca254c9e8263c51fccf3f3ed41e481ac9 WatchSource:0}: Error finding container 7ad8aa3e3ab54f2971b49fe26be5d90ca254c9e8263c51fccf3f3ed41e481ac9: Status 404 returned error can't find the container with id 7ad8aa3e3ab54f2971b49fe26be5d90ca254c9e8263c51fccf3f3ed41e481ac9 Nov 02 18:27:42.160273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:42.160228 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:42.160371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:42.160309 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:42.160371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:42.160338 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:27:43.129083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:43.129035 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" event={"ID":"3f84129a-bb35-4c76-9db8-bb15ea5fcd36","Type":"ContainerStarted","Data":"fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4"} Nov 02 18:27:43.129327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:43.129105 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" event={"ID":"3f84129a-bb35-4c76-9db8-bb15ea5fcd36","Type":"ContainerStarted","Data":"7ad8aa3e3ab54f2971b49fe26be5d90ca254c9e8263c51fccf3f3ed41e481ac9"} Nov 02 18:27:43.322963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:43.322891 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podStartSLOduration=2.322869617 podStartE2EDuration="2.322869617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:43.299370355 +0000 UTC m=+3850.372897557" watchObservedRunningTime="2026-11-02 18:27:43.322869617 +0000 UTC m=+3850.396396821" Nov 02 18:27:43.723760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:43.719133 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:43.723760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:43.722045 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:43.723760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:43.722107 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:44.720243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:44.720204 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:44.720923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:44.720884 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:45.724235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:45.724193 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:45.727335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:45.726853 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:45.931103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:45.931060 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2"] Nov 02 18:27:45.931336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:45.931273 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" podUID="0ad3399d-fc04-4ceb-884f-588eb7e7ad96" containerName="maas-api" containerID="cri-o://e63147d8b06a4bd49207504134c46ad6ab87ebf891f2046b16767c342402b3f5" gracePeriod=30 Nov 02 18:27:46.194908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.194870 5647 generic.go:356] "Generic (PLEG): container finished" podID="0ad3399d-fc04-4ceb-884f-588eb7e7ad96" containerID="e63147d8b06a4bd49207504134c46ad6ab87ebf891f2046b16767c342402b3f5" exitCode=0 Nov 02 18:27:46.195262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.195241 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" event={"ID":"0ad3399d-fc04-4ceb-884f-588eb7e7ad96","Type":"ContainerDied","Data":"e63147d8b06a4bd49207504134c46ad6ab87ebf891f2046b16767c342402b3f5"} Nov 02 18:27:46.486573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.486542 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:27:46.582800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.582770 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs\") pod \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " Nov 02 18:27:46.582968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.582885 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5\" (UniqueName: \"kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5\") pod \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " Nov 02 18:27:46.582968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.582953 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\") pod \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " Nov 02 18:27:46.583094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.583027 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca\") pod \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\" (UID: \"0ad3399d-fc04-4ceb-884f-588eb7e7ad96\") " Nov 02 18:27:46.583372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.583348 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca" pod "0ad3399d-fc04-4ceb-884f-588eb7e7ad96" (UID: "0ad3399d-fc04-4ceb-884f-588eb7e7ad96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:27:46.583623 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.583606 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:46.586627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.585824 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs" pod "0ad3399d-fc04-4ceb-884f-588eb7e7ad96" (UID: "0ad3399d-fc04-4ceb-884f-588eb7e7ad96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:27:46.587007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.586983 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5" pod "0ad3399d-fc04-4ceb-884f-588eb7e7ad96" (UID: "0ad3399d-fc04-4ceb-884f-588eb7e7ad96"). InnerVolumeSpecName "kube-api-access-m42r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:27:46.587574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.587551 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls" pod "0ad3399d-fc04-4ceb-884f-588eb7e7ad96" (UID: "0ad3399d-fc04-4ceb-884f-588eb7e7ad96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:27:46.685188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.685107 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:46.685188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.685140 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m42r5\" (UniqueName: \"kubernetes.io/projected/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-kube-api-access-m42r5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:46.685188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.685151 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad3399d-fc04-4ceb-884f-588eb7e7ad96-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:27:46.721646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.721184 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:46.721646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:46.721251 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:47.229144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.229109 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" Nov 02 18:27:47.229553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.229171 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2" event={"ID":"0ad3399d-fc04-4ceb-884f-588eb7e7ad96","Type":"ContainerDied","Data":"a310116b7d3bab003c6a92411d987a8ad30cbc4669a6e91b86af4f8634ed9f60"} Nov 02 18:27:47.229553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.229261 5647 scope.go:122] "RemoveContainer" containerID="e63147d8b06a4bd49207504134c46ad6ab87ebf891f2046b16767c342402b3f5" Nov 02 18:27:47.245769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.245727 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:27:47.400408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.400372 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2"] Nov 02 18:27:47.433184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.433087 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-cd738b-64c85c9d8-8kml2"] Nov 02 18:27:47.729331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.729233 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:47.729608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:47.729575 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:48.717255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:48.717203 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:48.717904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:48.717857 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:49.082580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:49.082483 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad3399d-fc04-4ceb-884f-588eb7e7ad96" path="/var/lib/kubelet/pods/0ad3399d-fc04-4ceb-884f-588eb7e7ad96/volumes" Nov 02 18:27:49.718647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:49.718612 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-s4p96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Nov 02 18:27:49.719238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:49.718670 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Nov 02 18:27:51.599349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:51.599308 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:51.600916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:51.600881 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:27:51.601967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:27:51.601942 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:28:05.948543 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:28:05.948471 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:28:09.699675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:09.699624 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc"] Nov 02 18:28:09.700185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:09.699904 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" containerID="cri-o://a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d" gracePeriod=30 Nov 02 18:28:13.440186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:13.440099 5647 patch_prober.go:29] interesting pod/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:28:13.440186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:13.440163 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:28:14.972715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:14.972646 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:28:15.121850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.121814 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.121868 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.121911 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.121953 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.121979 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122066 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122125 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122211 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122227 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:15.122309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122248 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket\") pod \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\" (UID: \"cd35a2c9-eccb-4d26-bcb0-22628179fda8\") " Nov 02 18:28:15.122587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122387 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:15.122587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122400 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:28:15.122587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122516 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:15.122725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122668 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:15.122985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122962 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.122985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.122986 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.123199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.123001 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.123199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.123014 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.123199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.123027 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.124892 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.124866 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:28:15.125027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.124901 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:15.125114 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.125075 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:28:15.125424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.125404 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59" pod "cd35a2c9-eccb-4d26-bcb0-22628179fda8" (UID: "cd35a2c9-eccb-4d26-bcb0-22628179fda8"). InnerVolumeSpecName "kube-api-access-r8j59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:28:15.224609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.224513 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.224609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.224554 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.224609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.224571 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8j59\" (UniqueName: \"kubernetes.io/projected/cd35a2c9-eccb-4d26-bcb0-22628179fda8-kube-api-access-r8j59\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.224609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.224587 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd35a2c9-eccb-4d26-bcb0-22628179fda8-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:28:15.450428 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.450394 5647 generic.go:356] "Generic (PLEG): container finished" podID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerID="a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d" exitCode=0 Nov 02 18:28:15.450605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.450457 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" Nov 02 18:28:15.450605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.450478 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" event={"ID":"cd35a2c9-eccb-4d26-bcb0-22628179fda8","Type":"ContainerDied","Data":"a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d"} Nov 02 18:28:15.450605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.450530 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc" event={"ID":"cd35a2c9-eccb-4d26-bcb0-22628179fda8","Type":"ContainerDied","Data":"cbdf498c64676320c413eb1104aef07fba3233a7baa65a2400c96cdf9b7b4bc2"} Nov 02 18:28:15.450605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.450551 5647 scope.go:122] "RemoveContainer" containerID="a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d" Nov 02 18:28:15.461521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.461499 5647 scope.go:122] "RemoveContainer" containerID="3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5" Nov 02 18:28:15.471698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.471678 5647 scope.go:122] "RemoveContainer" containerID="a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d" Nov 02 18:28:15.472022 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:28:15.471996 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d\": container with ID starting with a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d not found: ID does not exist" containerID="a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d" Nov 02 18:28:15.472094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.472044 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d"} err="failed to get container status \"a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d\": rpc error: code = NotFound desc = could not find container \"a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d\": container with ID starting with a37520d5099a7cda251f322764b2e634621b1ee66f4165a657edb2f235d68b3d not found: ID does not exist" Nov 02 18:28:15.472094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.472063 5647 scope.go:122] "RemoveContainer" containerID="3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5" Nov 02 18:28:15.472320 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:28:15.472295 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5\": container with ID starting with 3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5 not found: ID does not exist" containerID="3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5" Nov 02 18:28:15.472382 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.472331 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5"} err="failed to get container status \"3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5\": rpc error: code = NotFound desc = could not find container \"3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5\": container with ID starting with 3f8cbf6e6a6b803b6c801c7183b80549381c454faa7e60b91e9073fec9b304c5 not found: ID does not exist" Nov 02 18:28:15.625523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.625485 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc"] Nov 02 18:28:15.628269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:15.628247 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd9ae6-openshift-default-5448546c49-4k8dc"] Nov 02 18:28:17.082153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:17.082114 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" path="/var/lib/kubelet/pods/cd35a2c9-eccb-4d26-bcb0-22628179fda8/volumes" Nov 02 18:28:25.286947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:25.286881 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:28:34.510840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.510810 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:28:34.519059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.519027 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:28:34.525835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.525810 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz"] Nov 02 18:28:34.526467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.526444 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" Nov 02 18:28:34.526467 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.526465 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" Nov 02 18:28:34.526670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.526497 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0ad3399d-fc04-4ceb-884f-588eb7e7ad96" containerName="maas-api" Nov 02 18:28:34.526670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.526503 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad3399d-fc04-4ceb-884f-588eb7e7ad96" containerName="maas-api" Nov 02 18:28:34.526670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.526624 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" Nov 02 18:28:34.526670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.526638 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad3399d-fc04-4ceb-884f-588eb7e7ad96" containerName="maas-api" Nov 02 18:28:34.532103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.532083 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.585846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.585806 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-662a80-serving-cert\"" Nov 02 18:28:34.639145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.639101 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz"] Nov 02 18:28:34.661185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.661147 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.661393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.661200 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.661393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.661376 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rlr\" (UniqueName: \"kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.661536 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.661512 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.762924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.762817 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.762924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.762886 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.763154 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.762953 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rlr\" (UniqueName: \"kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.763154 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:28:34.762975 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-serving-cert: secret "maas-api-e2e-inf-a-662a80-serving-cert" not found Nov 02 18:28:34.763154 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.763004 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.763154 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:28:34.763038 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls podName:f151ed30-8d9c-40e2-8425-bb4b4c17b624 nodeName:}" failed. No retries permitted until 2026-11-02 18:28:35.263022472 +0000 UTC m=+3902.336549653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls") pod "maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" (UID: "f151ed30-8d9c-40e2-8425-bb4b4c17b624") : secret "maas-api-e2e-inf-a-662a80-serving-cert" not found Nov 02 18:28:34.763625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.763607 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.765523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.765498 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:34.825408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:34.825373 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rlr\" (UniqueName: \"kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:35.274773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:35.274729 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:35.277409 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:35.277375 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\") pod \"maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:35.447254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:35.447223 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:35.825895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:35.825867 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz"] Nov 02 18:28:35.827283 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:28:35.827250 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf151ed30_8d9c_40e2_8425_bb4b4c17b624.slice/crio-6d9773d08ba7c7781dcef55f36707fa5787a3f7b70e7424a4694c7d462ba891f WatchSource:0}: Error finding container 6d9773d08ba7c7781dcef55f36707fa5787a3f7b70e7424a4694c7d462ba891f: Status 404 returned error can't find the container with id 6d9773d08ba7c7781dcef55f36707fa5787a3f7b70e7424a4694c7d462ba891f Nov 02 18:28:35.828952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:35.828933 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:28:35.947719 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:28:35.947683 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:28:36.647085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:36.647046 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" event={"ID":"f151ed30-8d9c-40e2-8425-bb4b4c17b624","Type":"ContainerStarted","Data":"6d9773d08ba7c7781dcef55f36707fa5787a3f7b70e7424a4694c7d462ba891f"} Nov 02 18:28:37.679953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:37.679169 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" event={"ID":"f151ed30-8d9c-40e2-8425-bb4b4c17b624","Type":"ContainerStarted","Data":"048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b"} Nov 02 18:28:37.679953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:37.679798 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:28:37.841330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:37.841277 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" podStartSLOduration=2.8621897389999997 podStartE2EDuration="3.841260423s" totalImagesPullingTime="979.070684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:37.841166934 +0000 UTC m=+3904.914694138" watchObservedRunningTime="2026-11-02 18:28:37.841260423 +0000 UTC m=+3904.914787625" Nov 02 18:28:43.692113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:28:43.692079 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:29:03.985718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:03.985676 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:29:05.948296 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:29:05.948255 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:29:12.641716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:12.641683 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl"] Nov 02 18:29:12.642165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:12.641929 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" podUID="004cb34f-80f6-44fa-bc55-6c6e4341bd27" containerName="maas-api" containerID="cri-o://f43b45144203a947de164c7d433966b4df864bb04a22cc7dc5ca420ab943afd2" gracePeriod=30 Nov 02 18:29:12.969783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:12.969718 5647 generic.go:356] "Generic (PLEG): container finished" podID="004cb34f-80f6-44fa-bc55-6c6e4341bd27" containerID="f43b45144203a947de164c7d433966b4df864bb04a22cc7dc5ca420ab943afd2" exitCode=0 Nov 02 18:29:12.969783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:12.969766 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" event={"ID":"004cb34f-80f6-44fa-bc55-6c6e4341bd27","Type":"ContainerDied","Data":"f43b45144203a947de164c7d433966b4df864bb04a22cc7dc5ca420ab943afd2"} Nov 02 18:29:12.989189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:12.989163 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:29:13.007020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.006992 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs\") pod \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " Nov 02 18:29:13.007178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.007106 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca\") pod \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " Nov 02 18:29:13.007178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.007159 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\") pod \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " Nov 02 18:29:13.007278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.007263 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m\" (UniqueName: \"kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m\") pod \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\" (UID: \"004cb34f-80f6-44fa-bc55-6c6e4341bd27\") " Nov 02 18:29:13.007452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.007431 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca" pod "004cb34f-80f6-44fa-bc55-6c6e4341bd27" (UID: "004cb34f-80f6-44fa-bc55-6c6e4341bd27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:13.007849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.007828 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/004cb34f-80f6-44fa-bc55-6c6e4341bd27-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:13.009345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.009319 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs" pod "004cb34f-80f6-44fa-bc55-6c6e4341bd27" (UID: "004cb34f-80f6-44fa-bc55-6c6e4341bd27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:13.009449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.009341 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls" pod "004cb34f-80f6-44fa-bc55-6c6e4341bd27" (UID: "004cb34f-80f6-44fa-bc55-6c6e4341bd27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:29:13.009816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.009798 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m" pod "004cb34f-80f6-44fa-bc55-6c6e4341bd27" (UID: "004cb34f-80f6-44fa-bc55-6c6e4341bd27"). InnerVolumeSpecName "kube-api-access-b792m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:13.111376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.111343 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b792m\" (UniqueName: \"kubernetes.io/projected/004cb34f-80f6-44fa-bc55-6c6e4341bd27-kube-api-access-b792m\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:13.111376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.111376 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:13.111652 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.111388 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/004cb34f-80f6-44fa-bc55-6c6e4341bd27-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:13.977661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.977629 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" Nov 02 18:29:13.977661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.977649 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl" event={"ID":"004cb34f-80f6-44fa-bc55-6c6e4341bd27","Type":"ContainerDied","Data":"0e2af5d5c2015210dd4cd9534ae91ffb29b0d3e4a9d2f71d7995a1224287ca36"} Nov 02 18:29:13.978209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:13.977710 5647 scope.go:122] "RemoveContainer" containerID="f43b45144203a947de164c7d433966b4df864bb04a22cc7dc5ca420ab943afd2" Nov 02 18:29:14.147127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:14.147094 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl"] Nov 02 18:29:14.150720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:14.150683 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a72f55-8ffcf7cc8-qw9zl"] Nov 02 18:29:15.017401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:15.017364 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24"] Nov 02 18:29:15.017931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:15.017559 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" containerID="cri-o://5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8" gracePeriod=30 Nov 02 18:29:15.081394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:15.081358 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004cb34f-80f6-44fa-bc55-6c6e4341bd27" path="/var/lib/kubelet/pods/004cb34f-80f6-44fa-bc55-6c6e4341bd27/volumes" Nov 02 18:29:16.613299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.613257 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d"] Nov 02 18:29:16.613967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.613951 5647 cpu_manager.go:414] "containerMap: removing container" podUID="004cb34f-80f6-44fa-bc55-6c6e4341bd27" containerName="maas-api" Nov 02 18:29:16.614026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.613968 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004cb34f-80f6-44fa-bc55-6c6e4341bd27" containerName="maas-api" Nov 02 18:29:16.614026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.613993 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" Nov 02 18:29:16.614026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.613998 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" Nov 02 18:29:16.614135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.614091 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd35a2c9-eccb-4d26-bcb0-22628179fda8" containerName="istio-proxy" Nov 02 18:29:16.614135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.614108 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="004cb34f-80f6-44fa-bc55-6c6e4341bd27" containerName="maas-api" Nov 02 18:29:16.618745 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.618714 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.672609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.672572 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57pq\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.672874 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.672702 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.672874 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.672845 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.673002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.672880 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.673002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.672943 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.673002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.672973 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.673175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.673109 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.673235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.673180 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.673295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.673260 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.710555 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.710527 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c70712-openshift-default-dockercfg-zzmdw\"" Nov 02 18:29:16.726543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.726506 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d"] Nov 02 18:29:16.817244 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.817204 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.817579 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.817557 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.817816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.817797 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57pq\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.818013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.817996 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.818174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.818157 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.818294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.818277 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.818417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.818400 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.821336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.821309 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.822048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.822022 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.823321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.819847 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.823479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.820064 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.823592 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.819417 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.823692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.819671 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.823826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.823107 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.832070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.832038 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.832577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.832527 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.877264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.877184 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57pq\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.877414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.877265 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token\") pod \"e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:16.933281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:16.933246 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:17.417906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:17.417871 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d"] Nov 02 18:29:17.418063 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:29:17.417885 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a60368_c286_4b63_b463_daca81f664eb.slice/crio-42e509ea58a0be2e7eed4539016a52d9fd0b9d3aa3e29e80cb37e13804c39dc6 WatchSource:0}: Error finding container 42e509ea58a0be2e7eed4539016a52d9fd0b9d3aa3e29e80cb37e13804c39dc6: Status 404 returned error can't find the container with id 42e509ea58a0be2e7eed4539016a52d9fd0b9d3aa3e29e80cb37e13804c39dc6 Nov 02 18:29:17.420825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:17.420792 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:29:17.420903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:17.420857 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:29:17.420903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:17.420887 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:29:18.012175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:18.012137 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" event={"ID":"05a60368-c286-4b63-b463-daca81f664eb","Type":"ContainerStarted","Data":"4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1"} Nov 02 18:29:18.012590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:18.012202 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" event={"ID":"05a60368-c286-4b63-b463-daca81f664eb","Type":"ContainerStarted","Data":"42e509ea58a0be2e7eed4539016a52d9fd0b9d3aa3e29e80cb37e13804c39dc6"} Nov 02 18:29:18.134838 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:18.134774 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podStartSLOduration=2.134757371 podStartE2EDuration="2.134757371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:18.132998537 +0000 UTC m=+3945.206525776" watchObservedRunningTime="2026-11-02 18:29:18.134757371 +0000 UTC m=+3945.208284568" Nov 02 18:29:18.934064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:18.934029 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:18.940197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:18.940167 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:19.019184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:19.019154 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:19.020157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:19.020138 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:29:20.288122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.288055 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:29:20.388103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388071 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388114 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388202 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388226 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388308 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388337 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388373 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:20.388469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388392 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388459 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388824 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388537 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket\") pod \"e8df77cd-7619-4d45-ad37-7946076ed686\" (UID: \"e8df77cd-7619-4d45-ad37-7946076ed686\") " Nov 02 18:29:20.388824 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388653 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:20.388824 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.388809 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:20.389166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389098 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:29:20.389278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389176 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:20.389450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389430 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.389527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389452 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.389527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389463 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8df77cd-7619-4d45-ad37-7946076ed686-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.389527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389472 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.389527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.389482 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.390704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.390681 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:29:20.390825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.390759 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "kube-api-access-tnftg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:20.391103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.391075 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:29:20.391206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.391150 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy" pod "e8df77cd-7619-4d45-ad37-7946076ed686" (UID: "e8df77cd-7619-4d45-ad37-7946076ed686"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:29:20.490883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.490848 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8df77cd-7619-4d45-ad37-7946076ed686-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.490883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.490876 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.490883 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.490886 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8df77cd-7619-4d45-ad37-7946076ed686-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:20.491128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:20.490895 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnftg\" (UniqueName: \"kubernetes.io/projected/e8df77cd-7619-4d45-ad37-7946076ed686-kube-api-access-tnftg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:29:21.033533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.033494 5647 generic.go:356] "Generic (PLEG): container finished" podID="e8df77cd-7619-4d45-ad37-7946076ed686" containerID="5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8" exitCode=0 Nov 02 18:29:21.033761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.033544 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" Nov 02 18:29:21.033761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.033583 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" event={"ID":"e8df77cd-7619-4d45-ad37-7946076ed686","Type":"ContainerDied","Data":"5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8"} Nov 02 18:29:21.033761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.033641 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24" event={"ID":"e8df77cd-7619-4d45-ad37-7946076ed686","Type":"ContainerDied","Data":"fae2c735fac819a47c8f382817252ef6cf6f8ac7c5b7d405b6921f6ed5c5afed"} Nov 02 18:29:21.033761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.033674 5647 scope.go:122] "RemoveContainer" containerID="5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8" Nov 02 18:29:21.046464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.046443 5647 scope.go:122] "RemoveContainer" containerID="15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8" Nov 02 18:29:21.056709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.056359 5647 scope.go:122] "RemoveContainer" containerID="5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8" Nov 02 18:29:21.056963 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:29:21.056748 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8\": container with ID starting with 5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8 not found: ID does not exist" containerID="5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8" Nov 02 18:29:21.056963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.056786 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8"} err="failed to get container status \"5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8\": rpc error: code = NotFound desc = could not find container \"5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8\": container with ID starting with 5a627820fc1a028f51c4aee831a5acd33d478f71143e0f01f4f613139e0183f8 not found: ID does not exist" Nov 02 18:29:21.056963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.056815 5647 scope.go:122] "RemoveContainer" containerID="15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8" Nov 02 18:29:21.057176 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:29:21.057144 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8\": container with ID starting with 15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8 not found: ID does not exist" containerID="15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8" Nov 02 18:29:21.057240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.057183 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8"} err="failed to get container status \"15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8\": rpc error: code = NotFound desc = could not find container \"15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8\": container with ID starting with 15c49169aec5c7ccc196073c69b412ee5b74d676b1d1c2e61070ec49746cb0b8 not found: ID does not exist" Nov 02 18:29:21.209404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.209369 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24"] Nov 02 18:29:21.260247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:21.260214 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a72f55-openshift-default-9d5c6d94f-8kd24"] Nov 02 18:29:23.081644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:23.081610 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" path="/var/lib/kubelet/pods/e8df77cd-7619-4d45-ad37-7946076ed686/volumes" Nov 02 18:29:24.930480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.930373 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls"] Nov 02 18:29:24.951341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.930966 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" Nov 02 18:29:24.951341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.930978 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" Nov 02 18:29:24.951341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.931096 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" Nov 02 18:29:24.951341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.931107 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" Nov 02 18:29:24.951341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.936056 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:24.992220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:24.992182 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls"] Nov 02 18:29:25.039704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.039677 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c70712\"" Nov 02 18:29:25.067746 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.067679 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctj44\" (UniqueName: \"kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44\") pod \"payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.068333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.068298 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.170093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.170042 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctj44\" (UniqueName: \"kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44\") pod \"payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.170616 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.170591 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.172791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.172767 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.177600 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.177571 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp"] Nov 02 18:29:25.178786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.178763 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" Nov 02 18:29:25.178921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.178907 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8df77cd-7619-4d45-ad37-7946076ed686" containerName="istio-proxy" Nov 02 18:29:25.189008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.188936 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp"] Nov 02 18:29:25.189318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.189300 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.246029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.245981 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c70712-dockercfg-vfwrh\"" Nov 02 18:29:25.256535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.256496 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctj44\" (UniqueName: \"kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44\") pod \"payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.272651 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.272613 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.273079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.273056 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bvm\" (UniqueName: \"kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm\") pod \"payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.374706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.374663 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bvm\" (UniqueName: \"kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm\") pod \"payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.374955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.374826 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.375570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.375548 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.448181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.448102 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bvm\" (UniqueName: \"kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm\") pod \"payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.551170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.551135 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:25.573233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.573202 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:25.943165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.943133 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls"] Nov 02 18:29:25.959217 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:29:25.959182 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4be0196_409c_4a35_841d_eb59afc69c26.slice/crio-e0b0e2820934e52abd8edc3b737575e4ac58ea53cf7dbb7d33190969972f683f WatchSource:0}: Error finding container e0b0e2820934e52abd8edc3b737575e4ac58ea53cf7dbb7d33190969972f683f: Status 404 returned error can't find the container with id e0b0e2820934e52abd8edc3b737575e4ac58ea53cf7dbb7d33190969972f683f Nov 02 18:29:25.995453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:25.995162 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp"] Nov 02 18:29:25.996124 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:29:25.996083 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97ba79d_5e3e_44f5_8833_96522d933e16.slice/crio-de0d751784688d76f87454e2ed101eb74f923f89f83a0eb63ac9a4a26fc642c1 WatchSource:0}: Error finding container de0d751784688d76f87454e2ed101eb74f923f89f83a0eb63ac9a4a26fc642c1: Status 404 returned error can't find the container with id de0d751784688d76f87454e2ed101eb74f923f89f83a0eb63ac9a4a26fc642c1 Nov 02 18:29:26.080819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.080783 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" event={"ID":"f97ba79d-5e3e-44f5-8833-96522d933e16","Type":"ContainerStarted","Data":"7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42"} Nov 02 18:29:26.081019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.080848 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" event={"ID":"f97ba79d-5e3e-44f5-8833-96522d933e16","Type":"ContainerStarted","Data":"de0d751784688d76f87454e2ed101eb74f923f89f83a0eb63ac9a4a26fc642c1"} Nov 02 18:29:26.081019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.080879 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:26.082716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.082686 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" event={"ID":"d4be0196-409c-4a35-841d-eb59afc69c26","Type":"ContainerStarted","Data":"33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f"} Nov 02 18:29:26.082871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.082756 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:26.082871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.082781 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" event={"ID":"d4be0196-409c-4a35-841d-eb59afc69c26","Type":"ContainerStarted","Data":"e0b0e2820934e52abd8edc3b737575e4ac58ea53cf7dbb7d33190969972f683f"} Nov 02 18:29:26.200714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:26.200417 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" podStartSLOduration=1.200398693 podStartE2EDuration="1.200398693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:26.198464698 +0000 UTC m=+3953.271991914" watchObservedRunningTime="2026-11-02 18:29:26.200398693 +0000 UTC m=+3953.273925899" Nov 02 18:29:35.950103 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:29:35.950063 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:29:37.132460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:37.132436 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:29:37.146218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:37.146191 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:29:37.249175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:37.249116 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" podStartSLOduration=13.249100814 podStartE2EDuration="13.249100814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:26.334802782 +0000 UTC m=+3953.408329986" watchObservedRunningTime="2026-11-02 18:29:37.249100814 +0000 UTC m=+3964.322628092" Nov 02 18:29:38.308848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:38.308725 5647 generic.go:356] "Generic (PLEG): container finished" podID="aafb611c-d65f-447d-b126-881d5e3749f4" containerID="f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e" exitCode=0 Nov 02 18:29:38.308848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:38.308805 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" event={"ID":"aafb611c-d65f-447d-b126-881d5e3749f4","Type":"ContainerDied","Data":"f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e"} Nov 02 18:29:38.309325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:38.309239 5647 scope.go:122] "RemoveContainer" containerID="f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e" Nov 02 18:29:38.311041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:38.311007 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:29:38.311155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:38.311080 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:29:38.311155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:38.311115 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:29:39.334390 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:39.334347 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" event={"ID":"aafb611c-d65f-447d-b126-881d5e3749f4","Type":"ContainerStarted","Data":"67b49d009f2af1329f0a3a2a883af240d2994160d049af0d908da91289c69195"} Nov 02 18:29:40.062238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:40.062200 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:29:40.062437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:40.062417 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 02 18:29:40.062490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:40.062465 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 02 18:29:41.063083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:41.063041 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 02 18:29:41.063577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:41.063111 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 02 18:29:42.062303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:42.062266 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 02 18:29:42.062518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:42.062326 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 02 18:29:43.062923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:43.062883 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 02 18:29:43.063338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:43.062960 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 02 18:29:44.062429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:44.062393 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 02 18:29:44.062672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:44.062460 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 02 18:29:45.067670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:45.067637 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:29:45.068204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:45.067955 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:29:45.068926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:45.068910 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:29:58.716065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:29:58.716030 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:30:00.165939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.165896 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc"] Nov 02 18:30:00.171694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.171663 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596"] Nov 02 18:30:00.171934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.171853 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.175955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.175931 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g"] Nov 02 18:30:00.176118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.176010 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.179990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.179967 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.186343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.186315 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.186474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.186436 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd92w\" (UniqueName: \"kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w\") pod \"maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.186518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.186471 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.186761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.186715 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dqwk\" (UniqueName: \"kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.186863 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.186845 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.186905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.186880 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fl7\" (UniqueName: \"kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7\") pod \"maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.217204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.217166 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x"] Nov 02 18:30:00.222372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.222346 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j"] Nov 02 18:30:00.222523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.222468 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.226581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.226559 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.232499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.232475 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc"] Nov 02 18:30:00.235194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.235168 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg"] Nov 02 18:30:00.239638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.239565 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.264893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.264842 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:30:00.272865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.272821 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596"] Nov 02 18:30:00.284754 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.279283 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g"] Nov 02 18:30:00.287763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.285423 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j"] Nov 02 18:30:00.297728 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.297669 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrpk\" (UniqueName: \"kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.297728 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.297775 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dqwk\" (UniqueName: \"kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.298104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.297896 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.298104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.297926 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.298252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298151 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fl7\" (UniqueName: \"kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7\") pod \"maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.298252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298197 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.298252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298235 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.298431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298349 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjqc\" (UniqueName: \"kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.298431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298421 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd92w\" (UniqueName: \"kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w\") pod \"maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.298553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298458 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.298553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298508 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.298553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.298542 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvvf\" (UniqueName: \"kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.299407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.299374 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.299789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.299600 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.301010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.300980 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.324447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.324395 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg"] Nov 02 18:30:00.335547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.335510 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x"] Nov 02 18:30:00.358532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.358485 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fl7\" (UniqueName: \"kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7\") pod \"maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.375125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.375085 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd92w\" (UniqueName: \"kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w\") pod \"maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.375345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.375322 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dqwk\" (UniqueName: \"kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk\") pod \"maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.401401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.401367 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.401602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.401422 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.401602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.401450 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjqc\" (UniqueName: \"kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.401602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.401498 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.401602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.401523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvvf\" (UniqueName: \"kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.401846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.401652 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrpk\" (UniqueName: \"kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.402157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.402136 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.402239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.402222 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.402278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.402221 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.460063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.459987 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvvf\" (UniqueName: \"kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.460354 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.460330 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrpk\" (UniqueName: \"kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.460425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.460410 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjqc\" (UniqueName: \"kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.487641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.487599 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:00.497115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.497088 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:00.504217 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.504183 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:00.538811 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.538715 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:00.547089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.547061 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:00.573687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.567065 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:00.886819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.886788 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc"] Nov 02 18:30:00.887978 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:00.887950 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de5b946_4dfc_43a8_a160_afd9a4e19d77.slice/crio-bc6d180e4bdfdbea70801b821a978fc34d222e918efeed1143bb846be69a547d WatchSource:0}: Error finding container bc6d180e4bdfdbea70801b821a978fc34d222e918efeed1143bb846be69a547d: Status 404 returned error can't find the container with id bc6d180e4bdfdbea70801b821a978fc34d222e918efeed1143bb846be69a547d Nov 02 18:30:00.931662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.931633 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g"] Nov 02 18:30:00.933623 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:00.933590 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e3f5f4_f851_43cb_a130_be00f8b00552.slice/crio-8a819fbf44c847ae6bbcc27995c48a77e199c7dd8bdba1c90f55192df0668744 WatchSource:0}: Error finding container 8a819fbf44c847ae6bbcc27995c48a77e199c7dd8bdba1c90f55192df0668744: Status 404 returned error can't find the container with id 8a819fbf44c847ae6bbcc27995c48a77e199c7dd8bdba1c90f55192df0668744 Nov 02 18:30:00.934765 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:00.934706 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4a3b6d_cbc6_4cc1_b8aa_3dc1f9f28970.slice/crio-e9fc89b106692c27551be5befb97a8c4011cbf39be500c81d1cf9ee0bd398cf9 WatchSource:0}: Error finding container e9fc89b106692c27551be5befb97a8c4011cbf39be500c81d1cf9ee0bd398cf9: Status 404 returned error can't find the container with id e9fc89b106692c27551be5befb97a8c4011cbf39be500c81d1cf9ee0bd398cf9 Nov 02 18:30:00.935246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.935222 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596"] Nov 02 18:30:00.984453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.984430 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg"] Nov 02 18:30:00.985445 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:00.985417 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdac9796_0253_43bd_bb06_1f10bf38f253.slice/crio-19f781447b1abf6a3aaf0c20e618cbefda175cfb16e590f73e65ba21e5e864ed WatchSource:0}: Error finding container 19f781447b1abf6a3aaf0c20e618cbefda175cfb16e590f73e65ba21e5e864ed: Status 404 returned error can't find the container with id 19f781447b1abf6a3aaf0c20e618cbefda175cfb16e590f73e65ba21e5e864ed Nov 02 18:30:00.991688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:00.991662 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x"] Nov 02 18:30:00.992496 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:00.992457 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaec1de_cb74_495d_b2bb_510d9bb9624d.slice/crio-1c1b26b0b90c627994edc9dfeb9c9c940c2d20f2abb919a264d18da202b1be7d WatchSource:0}: Error finding container 1c1b26b0b90c627994edc9dfeb9c9c940c2d20f2abb919a264d18da202b1be7d: Status 404 returned error can't find the container with id 1c1b26b0b90c627994edc9dfeb9c9c940c2d20f2abb919a264d18da202b1be7d Nov 02 18:30:01.029864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.029832 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j"] Nov 02 18:30:01.119028 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:01.119000 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f54a78_6398_47bc_86af_d14b50459c61.slice/crio-6bcaee1b61e4c2aaa008b5a851363cce6c24c927257d8998fe0b38a6a11a02e4 WatchSource:0}: Error finding container 6bcaee1b61e4c2aaa008b5a851363cce6c24c927257d8998fe0b38a6a11a02e4: Status 404 returned error can't find the container with id 6bcaee1b61e4c2aaa008b5a851363cce6c24c927257d8998fe0b38a6a11a02e4 Nov 02 18:30:01.503772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.503713 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerStarted","Data":"8a819fbf44c847ae6bbcc27995c48a77e199c7dd8bdba1c90f55192df0668744"} Nov 02 18:30:01.506685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.506190 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerStarted","Data":"1c1b26b0b90c627994edc9dfeb9c9c940c2d20f2abb919a264d18da202b1be7d"} Nov 02 18:30:01.508297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.508264 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerStarted","Data":"dad6053a7e46fd66be7fede0486d0f2182f5eb6cdb5e1caca421883e1725f29e"} Nov 02 18:30:01.508449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.508326 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerStarted","Data":"e9fc89b106692c27551be5befb97a8c4011cbf39be500c81d1cf9ee0bd398cf9"} Nov 02 18:30:01.509994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.509967 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerStarted","Data":"b04642ad5da0f9de7ccbf0d639260ac11cc865b7f3fd1f432277842080a83f22"} Nov 02 18:30:01.510131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.510014 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerStarted","Data":"bc6d180e4bdfdbea70801b821a978fc34d222e918efeed1143bb846be69a547d"} Nov 02 18:30:01.511471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.511434 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerStarted","Data":"6bcaee1b61e4c2aaa008b5a851363cce6c24c927257d8998fe0b38a6a11a02e4"} Nov 02 18:30:01.512808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.512785 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerStarted","Data":"19f781447b1abf6a3aaf0c20e618cbefda175cfb16e590f73e65ba21e5e864ed"} Nov 02 18:30:01.624093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.624042 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" podStartSLOduration=1.624025494 podStartE2EDuration="1.624025494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.622966115 +0000 UTC m=+3988.696493331" watchObservedRunningTime="2026-11-02 18:30:01.624025494 +0000 UTC m=+3988.697552696" Nov 02 18:30:01.744604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:01.744496 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" podStartSLOduration=1.744481374 podStartE2EDuration="1.744481374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:01.742652679 +0000 UTC m=+3988.816179893" watchObservedRunningTime="2026-11-02 18:30:01.744481374 +0000 UTC m=+3988.818008576" Nov 02 18:30:02.354786 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:02.354746 5647 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdac9796_0253_43bd_bb06_1f10bf38f253.slice/crio-b3f48e46fdbcb1add3e2d4068bd9ea5466f3d0583f6d995d39051aa91776b734.scope\": RecentStats: unable to find data in memory cache]" Nov 02 18:30:02.521862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.521823 5647 generic.go:356] "Generic (PLEG): container finished" podID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerID="b04642ad5da0f9de7ccbf0d639260ac11cc865b7f3fd1f432277842080a83f22" exitCode=60 Nov 02 18:30:02.522321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.521919 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerDied","Data":"b04642ad5da0f9de7ccbf0d639260ac11cc865b7f3fd1f432277842080a83f22"} Nov 02 18:30:02.522321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.522167 5647 scope.go:122] "RemoveContainer" containerID="b04642ad5da0f9de7ccbf0d639260ac11cc865b7f3fd1f432277842080a83f22" Nov 02 18:30:02.523561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.523542 5647 generic.go:356] "Generic (PLEG): container finished" podID="05f54a78-6398-47bc-86af-d14b50459c61" containerID="92cb3fd33d00a613d860f455daa9cde70537c75f45bdec2de7fd79306fbec19f" exitCode=60 Nov 02 18:30:02.523663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.523648 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerDied","Data":"92cb3fd33d00a613d860f455daa9cde70537c75f45bdec2de7fd79306fbec19f"} Nov 02 18:30:02.524060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.524042 5647 scope.go:122] "RemoveContainer" containerID="92cb3fd33d00a613d860f455daa9cde70537c75f45bdec2de7fd79306fbec19f" Nov 02 18:30:02.525992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.525934 5647 generic.go:356] "Generic (PLEG): container finished" podID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerID="b3f48e46fdbcb1add3e2d4068bd9ea5466f3d0583f6d995d39051aa91776b734" exitCode=60 Nov 02 18:30:02.526059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.526023 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerDied","Data":"b3f48e46fdbcb1add3e2d4068bd9ea5466f3d0583f6d995d39051aa91776b734"} Nov 02 18:30:02.526282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.526264 5647 scope.go:122] "RemoveContainer" containerID="b3f48e46fdbcb1add3e2d4068bd9ea5466f3d0583f6d995d39051aa91776b734" Nov 02 18:30:02.527946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.527924 5647 generic.go:356] "Generic (PLEG): container finished" podID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerID="f75a7ddf42ed9670f8c1ca9fda41af7de76bcd52b7da029e87dd12942bf2d6b7" exitCode=60 Nov 02 18:30:02.528028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.527951 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerDied","Data":"f75a7ddf42ed9670f8c1ca9fda41af7de76bcd52b7da029e87dd12942bf2d6b7"} Nov 02 18:30:02.528198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.528182 5647 scope.go:122] "RemoveContainer" containerID="f75a7ddf42ed9670f8c1ca9fda41af7de76bcd52b7da029e87dd12942bf2d6b7" Nov 02 18:30:02.542472 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.530270 5647 generic.go:356] "Generic (PLEG): container finished" podID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerID="33639a96bdadd07ef99e888e91bb879cb5f7831d1c7429f96f40ea48a14580ad" exitCode=60 Nov 02 18:30:02.542472 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.530313 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerDied","Data":"33639a96bdadd07ef99e888e91bb879cb5f7831d1c7429f96f40ea48a14580ad"} Nov 02 18:30:02.542472 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.530844 5647 scope.go:122] "RemoveContainer" containerID="33639a96bdadd07ef99e888e91bb879cb5f7831d1c7429f96f40ea48a14580ad" Nov 02 18:30:02.544047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.544013 5647 generic.go:356] "Generic (PLEG): container finished" podID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerID="dad6053a7e46fd66be7fede0486d0f2182f5eb6cdb5e1caca421883e1725f29e" exitCode=60 Nov 02 18:30:02.544199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.544101 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerDied","Data":"dad6053a7e46fd66be7fede0486d0f2182f5eb6cdb5e1caca421883e1725f29e"} Nov 02 18:30:02.544480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:02.544463 5647 scope.go:122] "RemoveContainer" containerID="dad6053a7e46fd66be7fede0486d0f2182f5eb6cdb5e1caca421883e1725f29e" Nov 02 18:30:03.556236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.556152 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerStarted","Data":"72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da"} Nov 02 18:30:03.558644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.558616 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerStarted","Data":"73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3"} Nov 02 18:30:03.560578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.560554 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerStarted","Data":"bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8"} Nov 02 18:30:03.562700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.562670 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerStarted","Data":"38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4"} Nov 02 18:30:03.564766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.564712 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerStarted","Data":"1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241"} Nov 02 18:30:03.566632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.566608 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerStarted","Data":"876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97"} Nov 02 18:30:03.674766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.674694 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" podStartSLOduration=3.674680756 podStartE2EDuration="3.674680756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.672903978 +0000 UTC m=+3990.746431204" watchObservedRunningTime="2026-11-02 18:30:03.674680756 +0000 UTC m=+3990.748207959" Nov 02 18:30:03.790214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.790153 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" podStartSLOduration=3.790133993 podStartE2EDuration="3.790133993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.789110778 +0000 UTC m=+3990.862637982" watchObservedRunningTime="2026-11-02 18:30:03.790133993 +0000 UTC m=+3990.863661197" Nov 02 18:30:03.907630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:03.907556 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" podStartSLOduration=3.9075345820000003 podStartE2EDuration="3.907534582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:03.904512825 +0000 UTC m=+3990.978040029" watchObservedRunningTime="2026-11-02 18:30:03.907534582 +0000 UTC m=+3990.981061784" Nov 02 18:30:04.137772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.137699 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" podStartSLOduration=4.137681923 podStartE2EDuration="4.137681923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:04.136193523 +0000 UTC m=+3991.209720727" watchObservedRunningTime="2026-11-02 18:30:04.137681923 +0000 UTC m=+3991.211209126" Nov 02 18:30:04.575597 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.575496 5647 generic.go:356] "Generic (PLEG): container finished" podID="05f54a78-6398-47bc-86af-d14b50459c61" containerID="876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97" exitCode=60 Nov 02 18:30:04.576139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.575587 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerDied","Data":"876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97"} Nov 02 18:30:04.576139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.575668 5647 scope.go:122] "RemoveContainer" containerID="92cb3fd33d00a613d860f455daa9cde70537c75f45bdec2de7fd79306fbec19f" Nov 02 18:30:04.576139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.575927 5647 scope.go:122] "RemoveContainer" containerID="876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97" Nov 02 18:30:04.576304 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:04.576286 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j_odh-ai-gateway-infra(05f54a78-6398-47bc-86af-d14b50459c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" podUID="05f54a78-6398-47bc-86af-d14b50459c61" Nov 02 18:30:04.577867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.577848 5647 generic.go:356] "Generic (PLEG): container finished" podID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerID="72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da" exitCode=60 Nov 02 18:30:04.577989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.577928 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerDied","Data":"72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da"} Nov 02 18:30:04.578176 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.578161 5647 scope.go:122] "RemoveContainer" containerID="72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da" Nov 02 18:30:04.578481 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:04.578462 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg_odh-ai-gateway-infra(bdac9796-0253-43bd-bb06-1f10bf38f253)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" Nov 02 18:30:04.579850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.579828 5647 generic.go:356] "Generic (PLEG): container finished" podID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerID="73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3" exitCode=60 Nov 02 18:30:04.579959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.579856 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerDied","Data":"73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3"} Nov 02 18:30:04.580141 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.580123 5647 scope.go:122] "RemoveContainer" containerID="73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3" Nov 02 18:30:04.580426 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:04.580403 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596_odh-ai-gateway-infra(36e3f5f4-f851-43cb-a130-be00f8b00552)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" Nov 02 18:30:04.581934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.581909 5647 generic.go:356] "Generic (PLEG): container finished" podID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerID="bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8" exitCode=60 Nov 02 18:30:04.582014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.581989 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerDied","Data":"bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8"} Nov 02 18:30:04.582226 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.582210 5647 scope.go:122] "RemoveContainer" containerID="bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8" Nov 02 18:30:04.582543 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:04.582518 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x_odh-ai-gateway-infra(ecaec1de-cb74-495d-b2bb-510d9bb9624d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" Nov 02 18:30:04.584191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.584169 5647 generic.go:356] "Generic (PLEG): container finished" podID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerID="38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4" exitCode=60 Nov 02 18:30:04.584307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.584253 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerDied","Data":"38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4"} Nov 02 18:30:04.584476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.584454 5647 scope.go:122] "RemoveContainer" containerID="38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4" Nov 02 18:30:04.584712 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:04.584692 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g_odh-ai-gateway-infra(ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" Nov 02 18:30:04.586461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.586440 5647 generic.go:356] "Generic (PLEG): container finished" podID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerID="1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241" exitCode=60 Nov 02 18:30:04.586550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.586488 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerDied","Data":"1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241"} Nov 02 18:30:04.586700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.586684 5647 scope.go:122] "RemoveContainer" containerID="1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241" Nov 02 18:30:04.586939 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:04.586924 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc_odh-ai-gateway-infra(6de5b946-4dfc-43a8-a160-afd9a4e19d77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" Nov 02 18:30:04.590187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.590166 5647 scope.go:122] "RemoveContainer" containerID="b3f48e46fdbcb1add3e2d4068bd9ea5466f3d0583f6d995d39051aa91776b734" Nov 02 18:30:04.604143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.604122 5647 scope.go:122] "RemoveContainer" containerID="f75a7ddf42ed9670f8c1ca9fda41af7de76bcd52b7da029e87dd12942bf2d6b7" Nov 02 18:30:04.618107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.618083 5647 scope.go:122] "RemoveContainer" containerID="33639a96bdadd07ef99e888e91bb879cb5f7831d1c7429f96f40ea48a14580ad" Nov 02 18:30:04.631356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.631333 5647 scope.go:122] "RemoveContainer" containerID="dad6053a7e46fd66be7fede0486d0f2182f5eb6cdb5e1caca421883e1725f29e" Nov 02 18:30:04.644867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:04.644841 5647 scope.go:122] "RemoveContainer" containerID="b04642ad5da0f9de7ccbf0d639260ac11cc865b7f3fd1f432277842080a83f22" Nov 02 18:30:05.595965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:05.595939 5647 scope.go:122] "RemoveContainer" containerID="bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8" Nov 02 18:30:05.596449 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.596311 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x_odh-ai-gateway-infra(ecaec1de-cb74-495d-b2bb-510d9bb9624d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" Nov 02 18:30:05.597794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:05.597772 5647 scope.go:122] "RemoveContainer" containerID="38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4" Nov 02 18:30:05.598109 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.598086 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g_odh-ai-gateway-infra(ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" Nov 02 18:30:05.600781 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:05.600762 5647 scope.go:122] "RemoveContainer" containerID="1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241" Nov 02 18:30:05.601053 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.601034 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc_odh-ai-gateway-infra(6de5b946-4dfc-43a8-a160-afd9a4e19d77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" Nov 02 18:30:05.602369 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:05.602348 5647 scope.go:122] "RemoveContainer" containerID="876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97" Nov 02 18:30:05.602652 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.602628 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j_odh-ai-gateway-infra(05f54a78-6398-47bc-86af-d14b50459c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" podUID="05f54a78-6398-47bc-86af-d14b50459c61" Nov 02 18:30:05.604109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:05.604087 5647 scope.go:122] "RemoveContainer" containerID="72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da" Nov 02 18:30:05.604399 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.604374 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg_odh-ai-gateway-infra(bdac9796-0253-43bd-bb06-1f10bf38f253)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" Nov 02 18:30:05.605576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:05.605560 5647 scope.go:122] "RemoveContainer" containerID="73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3" Nov 02 18:30:05.605797 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.605780 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596_odh-ai-gateway-infra(36e3f5f4-f851-43cb-a130-be00f8b00552)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" Nov 02 18:30:05.948891 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:05.948800 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:30:07.404246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.404208 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp"] Nov 02 18:30:07.408662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.408639 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.473759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.468088 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6cf852-openshift-default-dockercfg-9j5vt\"" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.603848 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp"] Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606403 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606446 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5bd\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606477 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606510 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606613 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606644 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606676 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606769 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.608768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.606813 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.709648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.708667 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.710003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.709975 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.710767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.710342 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.711028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.710710 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.711171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.710963 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5bd\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.711350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.711324 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.711500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.711485 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.711992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.711596 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.712388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.712367 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.713008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.712688 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.713131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.713027 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.713749 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.713673 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.713935 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.713915 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.714110 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.714082 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.716421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.716281 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.717102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.717078 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.769920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.769881 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:07.770067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:07.769933 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5bd\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd\") pod \"e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:08.024014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.023927 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:08.459577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.459537 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp"] Nov 02 18:30:08.461169 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:08.461136 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afb4a63_964c_4903_8372_7f6cdd4022ff.slice/crio-7015c6ee631d44adbb9e4f42aaf9275434995b087ff582b9108d919e81449b86 WatchSource:0}: Error finding container 7015c6ee631d44adbb9e4f42aaf9275434995b087ff582b9108d919e81449b86: Status 404 returned error can't find the container with id 7015c6ee631d44adbb9e4f42aaf9275434995b087ff582b9108d919e81449b86 Nov 02 18:30:08.464061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.464025 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:30:08.464157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.464097 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:30:08.464157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.464127 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:30:08.703464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.703426 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" event={"ID":"6afb4a63-964c-4903-8372-7f6cdd4022ff","Type":"ContainerStarted","Data":"95f30523fabfcd5e091156060ed9a012b62b6dd75db0f89539b7f94832079f72"} Nov 02 18:30:08.703655 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.703489 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" event={"ID":"6afb4a63-964c-4903-8372-7f6cdd4022ff","Type":"ContainerStarted","Data":"7015c6ee631d44adbb9e4f42aaf9275434995b087ff582b9108d919e81449b86"} Nov 02 18:30:08.826236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:08.826124 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podStartSLOduration=1.8261084090000002 podStartE2EDuration="1.826108409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:08.823392741 +0000 UTC m=+3995.896919979" watchObservedRunningTime="2026-11-02 18:30:08.826108409 +0000 UTC m=+3995.899635611" Nov 02 18:30:09.031185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:09.031145 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:09.037432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:09.037383 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:09.710926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:09.710891 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:09.712024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:09.712000 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:30:14.074272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:14.074238 5647 scope.go:122] "RemoveContainer" containerID="1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241" Nov 02 18:30:14.745779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:14.745664 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerStarted","Data":"41f74ac6d3c0e3daed53f2fb08ad483fabbbab958d8e05844e52f09044445f28"} Nov 02 18:30:15.074474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.074448 5647 scope.go:122] "RemoveContainer" containerID="73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3" Nov 02 18:30:15.754463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.754428 5647 generic.go:356] "Generic (PLEG): container finished" podID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerID="41f74ac6d3c0e3daed53f2fb08ad483fabbbab958d8e05844e52f09044445f28" exitCode=60 Nov 02 18:30:15.754687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.754500 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerDied","Data":"41f74ac6d3c0e3daed53f2fb08ad483fabbbab958d8e05844e52f09044445f28"} Nov 02 18:30:15.754687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.754578 5647 scope.go:122] "RemoveContainer" containerID="1603cc4d8382d34a7f24ebb48b5c6fc191bb85ac8d270fecdbf15f71c9d45241" Nov 02 18:30:15.755021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.754999 5647 scope.go:122] "RemoveContainer" containerID="41f74ac6d3c0e3daed53f2fb08ad483fabbbab958d8e05844e52f09044445f28" Nov 02 18:30:15.755362 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:15.755337 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc_odh-ai-gateway-infra(6de5b946-4dfc-43a8-a160-afd9a4e19d77)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" Nov 02 18:30:15.756785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.756754 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerStarted","Data":"25bc5f5b432150cc2106e32d67c2ae24a42fc37f3c8ed59569f102dd8164b637"} Nov 02 18:30:15.874238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:15.874199 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc"] Nov 02 18:30:16.073926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.073900 5647 scope.go:122] "RemoveContainer" containerID="72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da" Nov 02 18:30:16.074776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.074723 5647 scope.go:122] "RemoveContainer" containerID="876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97" Nov 02 18:30:16.769820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.769785 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerStarted","Data":"ec2381dcab0ff80f259b18e05d2fe8478b2c76bcdf48a240c5f5880e4fa34b1a"} Nov 02 18:30:16.771928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.771901 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerStarted","Data":"feacaccecc91edd8e4808bf07551cc664a2742613c63d15dca58bc158d2d3c79"} Nov 02 18:30:16.773899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.773871 5647 generic.go:356] "Generic (PLEG): container finished" podID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerID="25bc5f5b432150cc2106e32d67c2ae24a42fc37f3c8ed59569f102dd8164b637" exitCode=60 Nov 02 18:30:16.773986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.773917 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerDied","Data":"25bc5f5b432150cc2106e32d67c2ae24a42fc37f3c8ed59569f102dd8164b637"} Nov 02 18:30:16.773986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.773966 5647 scope.go:122] "RemoveContainer" containerID="73ac9cad1599b44d5870ad1572a194f1d07392810193682ee7038a3035ddc5d3" Nov 02 18:30:16.774287 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.774269 5647 scope.go:122] "RemoveContainer" containerID="25bc5f5b432150cc2106e32d67c2ae24a42fc37f3c8ed59569f102dd8164b637" Nov 02 18:30:16.774588 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:16.774562 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596_odh-ai-gateway-infra(36e3f5f4-f851-43cb-a130-be00f8b00552)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" Nov 02 18:30:16.926052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:16.926026 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:17.025482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.025416 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596"] Nov 02 18:30:17.034943 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.034914 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7\" (UniqueName: \"kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7\") pod \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " Nov 02 18:30:17.035067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.035046 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca\") pod \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\" (UID: \"6de5b946-4dfc-43a8-a160-afd9a4e19d77\") " Nov 02 18:30:17.035449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.035424 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca" pod "6de5b946-4dfc-43a8-a160-afd9a4e19d77" (UID: "6de5b946-4dfc-43a8-a160-afd9a4e19d77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:17.035761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.035714 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6de5b946-4dfc-43a8-a160-afd9a4e19d77-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:17.037104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.037079 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7" pod "6de5b946-4dfc-43a8-a160-afd9a4e19d77" (UID: "6de5b946-4dfc-43a8-a160-afd9a4e19d77"). InnerVolumeSpecName "kube-api-access-r8fl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:17.074452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.074416 5647 scope.go:122] "RemoveContainer" containerID="38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4" Nov 02 18:30:17.074619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.074567 5647 scope.go:122] "RemoveContainer" containerID="bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8" Nov 02 18:30:17.137179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.137142 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8fl7\" (UniqueName: \"kubernetes.io/projected/6de5b946-4dfc-43a8-a160-afd9a4e19d77-kube-api-access-r8fl7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:17.783249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.783207 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerStarted","Data":"4ea02df5ad1383224c4ffc9d2e69d3af32ef82f56a7dae9a5bdca686b719ee16"} Nov 02 18:30:17.785360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.785334 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerStarted","Data":"268f2344e47f362dc0ddef94b62724b300fe46957943e4dd487ba4cb043f334e"} Nov 02 18:30:17.786895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.786870 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" Nov 02 18:30:17.787043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.786892 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc" event={"ID":"6de5b946-4dfc-43a8-a160-afd9a4e19d77","Type":"ContainerDied","Data":"bc6d180e4bdfdbea70801b821a978fc34d222e918efeed1143bb846be69a547d"} Nov 02 18:30:17.787043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.786951 5647 scope.go:122] "RemoveContainer" containerID="41f74ac6d3c0e3daed53f2fb08ad483fabbbab958d8e05844e52f09044445f28" Nov 02 18:30:17.789035 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.789009 5647 generic.go:356] "Generic (PLEG): container finished" podID="05f54a78-6398-47bc-86af-d14b50459c61" containerID="ec2381dcab0ff80f259b18e05d2fe8478b2c76bcdf48a240c5f5880e4fa34b1a" exitCode=60 Nov 02 18:30:17.789224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.789089 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerDied","Data":"ec2381dcab0ff80f259b18e05d2fe8478b2c76bcdf48a240c5f5880e4fa34b1a"} Nov 02 18:30:17.789514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.789494 5647 scope.go:122] "RemoveContainer" containerID="ec2381dcab0ff80f259b18e05d2fe8478b2c76bcdf48a240c5f5880e4fa34b1a" Nov 02 18:30:17.789847 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:17.789821 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j_odh-ai-gateway-infra(05f54a78-6398-47bc-86af-d14b50459c61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" podUID="05f54a78-6398-47bc-86af-d14b50459c61" Nov 02 18:30:17.791423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.791405 5647 generic.go:356] "Generic (PLEG): container finished" podID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerID="feacaccecc91edd8e4808bf07551cc664a2742613c63d15dca58bc158d2d3c79" exitCode=60 Nov 02 18:30:17.791524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.791466 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerDied","Data":"feacaccecc91edd8e4808bf07551cc664a2742613c63d15dca58bc158d2d3c79"} Nov 02 18:30:17.791791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.791776 5647 scope.go:122] "RemoveContainer" containerID="feacaccecc91edd8e4808bf07551cc664a2742613c63d15dca58bc158d2d3c79" Nov 02 18:30:17.792052 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:17.792032 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg_odh-ai-gateway-infra(bdac9796-0253-43bd-bb06-1f10bf38f253)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" Nov 02 18:30:17.799143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.799123 5647 scope.go:122] "RemoveContainer" containerID="876e26c4c92e4578a4bf34d6708c6e7c83234dbce6bf8e2113e2056f80127f97" Nov 02 18:30:17.813942 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:17.813916 5647 scope.go:122] "RemoveContainer" containerID="72ff71b33c8483cd7c5b8a108a097308beaad891b7d97aa35d53d2a950e787da" Nov 02 18:30:18.011725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.011693 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j"] Nov 02 18:30:18.039061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.039019 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:18.043208 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.043176 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc"] Nov 02 18:30:18.047480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.047454 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-546881-29894070-wxfwc"] Nov 02 18:30:18.058595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.058569 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk\" (UniqueName: \"kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk\") pod \"36e3f5f4-f851-43cb-a130-be00f8b00552\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " Nov 02 18:30:18.058826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.058810 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca\") pod \"36e3f5f4-f851-43cb-a130-be00f8b00552\" (UID: \"36e3f5f4-f851-43cb-a130-be00f8b00552\") " Nov 02 18:30:18.059304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.059233 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca" pod "36e3f5f4-f851-43cb-a130-be00f8b00552" (UID: "36e3f5f4-f851-43cb-a130-be00f8b00552"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:18.060915 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.060887 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk" pod "36e3f5f4-f851-43cb-a130-be00f8b00552" (UID: "36e3f5f4-f851-43cb-a130-be00f8b00552"). InnerVolumeSpecName "kube-api-access-4dqwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:18.161350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.161311 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dqwk\" (UniqueName: \"kubernetes.io/projected/36e3f5f4-f851-43cb-a130-be00f8b00552-kube-api-access-4dqwk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.161350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.161351 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e3f5f4-f851-43cb-a130-be00f8b00552-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:18.256382 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.256287 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg"] Nov 02 18:30:18.807266 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.807239 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" Nov 02 18:30:18.807398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.807242 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596" event={"ID":"36e3f5f4-f851-43cb-a130-be00f8b00552","Type":"ContainerDied","Data":"8a819fbf44c847ae6bbcc27995c48a77e199c7dd8bdba1c90f55192df0668744"} Nov 02 18:30:18.807398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.807383 5647 scope.go:122] "RemoveContainer" containerID="25bc5f5b432150cc2106e32d67c2ae24a42fc37f3c8ed59569f102dd8164b637" Nov 02 18:30:18.817925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.817841 5647 generic.go:356] "Generic (PLEG): container finished" podID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerID="4ea02df5ad1383224c4ffc9d2e69d3af32ef82f56a7dae9a5bdca686b719ee16" exitCode=60 Nov 02 18:30:18.818090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.817942 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerDied","Data":"4ea02df5ad1383224c4ffc9d2e69d3af32ef82f56a7dae9a5bdca686b719ee16"} Nov 02 18:30:18.818717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.818394 5647 scope.go:122] "RemoveContainer" containerID="4ea02df5ad1383224c4ffc9d2e69d3af32ef82f56a7dae9a5bdca686b719ee16" Nov 02 18:30:18.818927 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:18.818848 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x_odh-ai-gateway-infra(ecaec1de-cb74-495d-b2bb-510d9bb9624d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" Nov 02 18:30:18.820925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.820798 5647 scope.go:122] "RemoveContainer" containerID="bbe8be0ae4cd0949d106f5b2a11d8e77aac1dba6d776fcbea662edb85e119bf8" Nov 02 18:30:18.822160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.822133 5647 generic.go:356] "Generic (PLEG): container finished" podID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerID="268f2344e47f362dc0ddef94b62724b300fe46957943e4dd487ba4cb043f334e" exitCode=60 Nov 02 18:30:18.822289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.822197 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerDied","Data":"268f2344e47f362dc0ddef94b62724b300fe46957943e4dd487ba4cb043f334e"} Nov 02 18:30:18.822601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.822584 5647 scope.go:122] "RemoveContainer" containerID="268f2344e47f362dc0ddef94b62724b300fe46957943e4dd487ba4cb043f334e" Nov 02 18:30:18.822919 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:18.822898 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g_odh-ai-gateway-infra(ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" Nov 02 18:30:18.835350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.835324 5647 scope.go:122] "RemoveContainer" containerID="38da6f0bfeb5f4520f87abbd40c9343003b4b1fc8639ded82fed011f5034abc4" Nov 02 18:30:18.914011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.913972 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x"] Nov 02 18:30:18.996532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:18.996496 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:19.023024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.023000 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:19.074798 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.074703 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf\" (UniqueName: \"kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf\") pod \"05f54a78-6398-47bc-86af-d14b50459c61\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " Nov 02 18:30:19.074954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.074800 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca\") pod \"bdac9796-0253-43bd-bb06-1f10bf38f253\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " Nov 02 18:30:19.074954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.074831 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca\") pod \"05f54a78-6398-47bc-86af-d14b50459c61\" (UID: \"05f54a78-6398-47bc-86af-d14b50459c61\") " Nov 02 18:30:19.075080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.074967 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk\" (UniqueName: \"kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk\") pod \"bdac9796-0253-43bd-bb06-1f10bf38f253\" (UID: \"bdac9796-0253-43bd-bb06-1f10bf38f253\") " Nov 02 18:30:19.075222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.075198 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca" pod "bdac9796-0253-43bd-bb06-1f10bf38f253" (UID: "bdac9796-0253-43bd-bb06-1f10bf38f253"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:19.075305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.075244 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca" pod "05f54a78-6398-47bc-86af-d14b50459c61" (UID: "05f54a78-6398-47bc-86af-d14b50459c61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:19.075602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.075535 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdac9796-0253-43bd-bb06-1f10bf38f253-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.075672 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.075609 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f54a78-6398-47bc-86af-d14b50459c61-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.077118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.077093 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf" pod "05f54a78-6398-47bc-86af-d14b50459c61" (UID: "05f54a78-6398-47bc-86af-d14b50459c61"). InnerVolumeSpecName "kube-api-access-psvvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.077237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.077218 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk" pod "bdac9796-0253-43bd-bb06-1f10bf38f253" (UID: "bdac9796-0253-43bd-bb06-1f10bf38f253"). InnerVolumeSpecName "kube-api-access-9jrpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:19.082656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.082633 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" path="/var/lib/kubelet/pods/6de5b946-4dfc-43a8-a160-afd9a4e19d77/volumes" Nov 02 18:30:19.172420 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.172389 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g"] Nov 02 18:30:19.177347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.177320 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvvf\" (UniqueName: \"kubernetes.io/projected/05f54a78-6398-47bc-86af-d14b50459c61-kube-api-access-psvvf\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.177347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.177346 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jrpk\" (UniqueName: \"kubernetes.io/projected/bdac9796-0253-43bd-bb06-1f10bf38f253-kube-api-access-9jrpk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:19.391484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.391439 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596"] Nov 02 18:30:19.394990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.394959 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dc5689-29894070-gh596"] Nov 02 18:30:19.834316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.834290 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" Nov 02 18:30:19.834448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.834327 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j" event={"ID":"05f54a78-6398-47bc-86af-d14b50459c61","Type":"ContainerDied","Data":"6bcaee1b61e4c2aaa008b5a851363cce6c24c927257d8998fe0b38a6a11a02e4"} Nov 02 18:30:19.834448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.834392 5647 scope.go:122] "RemoveContainer" containerID="ec2381dcab0ff80f259b18e05d2fe8478b2c76bcdf48a240c5f5880e4fa34b1a" Nov 02 18:30:19.836044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.836021 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" Nov 02 18:30:19.836190 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.836053 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg" event={"ID":"bdac9796-0253-43bd-bb06-1f10bf38f253","Type":"ContainerDied","Data":"19f781447b1abf6a3aaf0c20e618cbefda175cfb16e590f73e65ba21e5e864ed"} Nov 02 18:30:19.861010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:19.860970 5647 scope.go:122] "RemoveContainer" containerID="feacaccecc91edd8e4808bf07551cc664a2742613c63d15dca58bc158d2d3c79" Nov 02 18:30:20.012759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.012722 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:20.041001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.040977 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:20.114553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.114524 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca\") pod \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " Nov 02 18:30:20.114753 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.114593 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca\") pod \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " Nov 02 18:30:20.114852 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.114749 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc\" (UniqueName: \"kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc\") pod \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\" (UID: \"ecaec1de-cb74-495d-b2bb-510d9bb9624d\") " Nov 02 18:30:20.115267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.114954 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w\" (UniqueName: \"kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w\") pod \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\" (UID: \"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970\") " Nov 02 18:30:20.115267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.114974 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca" pod "ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" (UID: "ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:20.115267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.114991 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca" pod "ecaec1de-cb74-495d-b2bb-510d9bb9624d" (UID: "ecaec1de-cb74-495d-b2bb-510d9bb9624d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:30:20.115668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.115646 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.115816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.115674 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecaec1de-cb74-495d-b2bb-510d9bb9624d-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.117326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.117299 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc" pod "ecaec1de-cb74-495d-b2bb-510d9bb9624d" (UID: "ecaec1de-cb74-495d-b2bb-510d9bb9624d"). InnerVolumeSpecName "kube-api-access-nnjqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:20.117437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.117424 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w" pod "ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" (UID: "ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970"). InnerVolumeSpecName "kube-api-access-nd92w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:30:20.196196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.196159 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg"] Nov 02 18:30:20.201437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.201400 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-f7aa2e-29894070-wvjqg"] Nov 02 18:30:20.217218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.217192 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd92w\" (UniqueName: \"kubernetes.io/projected/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970-kube-api-access-nd92w\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.217218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.217219 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnjqc\" (UniqueName: \"kubernetes.io/projected/ecaec1de-cb74-495d-b2bb-510d9bb9624d-kube-api-access-nnjqc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:30:20.312670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.312636 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j"] Nov 02 18:30:20.320469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.320434 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-0ad26a-29894070-rpx8j"] Nov 02 18:30:20.847833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.847801 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" Nov 02 18:30:20.848126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.847799 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x" event={"ID":"ecaec1de-cb74-495d-b2bb-510d9bb9624d","Type":"ContainerDied","Data":"1c1b26b0b90c627994edc9dfeb9c9c940c2d20f2abb919a264d18da202b1be7d"} Nov 02 18:30:20.848126 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.847947 5647 scope.go:122] "RemoveContainer" containerID="4ea02df5ad1383224c4ffc9d2e69d3af32ef82f56a7dae9a5bdca686b719ee16" Nov 02 18:30:20.849326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.849310 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" Nov 02 18:30:20.849419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.849356 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g" event={"ID":"ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970","Type":"ContainerDied","Data":"e9fc89b106692c27551be5befb97a8c4011cbf39be500c81d1cf9ee0bd398cf9"} Nov 02 18:30:20.858401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:20.858377 5647 scope.go:122] "RemoveContainer" containerID="268f2344e47f362dc0ddef94b62724b300fe46957943e4dd487ba4cb043f334e" Nov 02 18:30:21.019200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.019166 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x"] Nov 02 18:30:21.021831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.021806 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2a1bcc-29894070-4t76x"] Nov 02 18:30:21.081932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.081900 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f54a78-6398-47bc-86af-d14b50459c61" path="/var/lib/kubelet/pods/05f54a78-6398-47bc-86af-d14b50459c61/volumes" Nov 02 18:30:21.082363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.082350 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" path="/var/lib/kubelet/pods/36e3f5f4-f851-43cb-a130-be00f8b00552/volumes" Nov 02 18:30:21.082727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.082714 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" path="/var/lib/kubelet/pods/bdac9796-0253-43bd-bb06-1f10bf38f253/volumes" Nov 02 18:30:21.083410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.083397 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" path="/var/lib/kubelet/pods/ecaec1de-cb74-495d-b2bb-510d9bb9624d/volumes" Nov 02 18:30:21.131273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.131191 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g"] Nov 02 18:30:21.136480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:21.136456 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-740c43-29894070-jm92g"] Nov 02 18:30:23.081984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:23.081951 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" path="/var/lib/kubelet/pods/ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970/volumes" Nov 02 18:30:33.627704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.627660 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2"] Nov 02 18:30:33.628639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628615 5647 cpu_manager.go:414] "containerMap: removing container" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628643 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628662 5647 cpu_manager.go:414] "containerMap: removing container" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628673 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628687 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628694 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628707 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628714 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628728 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628753 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628766 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628774 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628784 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628790 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628798 5647 cpu_manager.go:414] "containerMap: removing container" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.628816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628804 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628830 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628837 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628848 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628855 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628870 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628876 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628902 5647 cpu_manager.go:414] "containerMap: removing container" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628909 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628918 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.628926 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629122 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629135 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629146 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629156 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629165 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629173 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629181 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629188 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629198 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629205 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629216 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629225 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629235 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629246 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:30:33.629675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.629255 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:30:33.635139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.635114 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.693759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.691928 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a51e27-openshift-default-dockercfg-92l7b\"" Nov 02 18:30:33.870613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.869660 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.870613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.869924 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.870613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.869979 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.870613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.870228 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.870613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.870284 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.871808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.870347 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.871808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.871559 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8tt\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.874188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.871642 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.874188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.872420 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.957177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.953994 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2"] Nov 02 18:30:33.975562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.975522 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.976349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.976326 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.977016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.976995 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.977215 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.977197 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.977380 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.977362 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.977519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.977504 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.977652 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.977638 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.977956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.977937 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8tt\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.978121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.978102 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.978931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.976214 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.979968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.979650 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.979968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.979923 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.980202 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.980171 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.980379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.980361 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.981049 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.981024 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:33.981565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:33.981521 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:34.036619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.036589 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8tt\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:34.036826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.036657 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token\") pod \"e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:34.249530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.249432 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:34.705333 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.705295 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2"] Nov 02 18:30:34.745052 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:30:34.745009 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf393f960_71c5_4ca2_bd2b_d7d1d49b9374.slice/crio-c9cb183a67c8ed48b85a78b23f0d73f7c983a5c8c1a5524af2ce5d8f297385c4 WatchSource:0}: Error finding container c9cb183a67c8ed48b85a78b23f0d73f7c983a5c8c1a5524af2ce5d8f297385c4: Status 404 returned error can't find the container with id c9cb183a67c8ed48b85a78b23f0d73f7c983a5c8c1a5524af2ce5d8f297385c4 Nov 02 18:30:34.748021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.747979 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:30:34.748134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.748058 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:30:34.748134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.748089 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:30:34.968026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.967920 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" event={"ID":"f393f960-71c5-4ca2-bd2b-d7d1d49b9374","Type":"ContainerStarted","Data":"5e5291c127e6cdfc5222440d52ab87e9c5c02cb256dd24cb47d08f8c49688d65"} Nov 02 18:30:34.968026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:34.967985 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" event={"ID":"f393f960-71c5-4ca2-bd2b-d7d1d49b9374","Type":"ContainerStarted","Data":"c9cb183a67c8ed48b85a78b23f0d73f7c983a5c8c1a5524af2ce5d8f297385c4"} Nov 02 18:30:35.090629 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:35.090574 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podStartSLOduration=2.090556338 podStartE2EDuration="2.090556338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:30:35.08922346 +0000 UTC m=+4022.162750690" watchObservedRunningTime="2026-11-02 18:30:35.090556338 +0000 UTC m=+4022.164083540" Nov 02 18:30:35.268812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:35.253130 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:35.268812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:35.258866 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:30:35.268812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:35.258929 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:30:35.948777 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:30:35.948721 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:30:36.255315 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:36.255241 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:36.983083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:36.983056 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:30:36.984038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:30:36.984002 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:31:05.947887 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:31:05.947836 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:31:11.344388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:11.344306 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:31:20.060036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:20.059971 5647 patch_prober.go:29] interesting pod/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:31:20.060615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:20.060084 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:31:23.403236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:23.403203 5647 generic.go:356] "Generic (PLEG): container finished" podID="05a60368-c286-4b63-b463-daca81f664eb" containerID="4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1" exitCode=0 Nov 02 18:31:23.403696 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:23.403274 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" event={"ID":"05a60368-c286-4b63-b463-daca81f664eb","Type":"ContainerDied","Data":"4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1"} Nov 02 18:31:23.403793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:23.403771 5647 scope.go:122] "RemoveContainer" containerID="4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1" Nov 02 18:31:23.405521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:23.405490 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:31:23.405640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:23.405562 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:31:23.405640 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:23.405597 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:31:24.432589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:24.432554 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" event={"ID":"05a60368-c286-4b63-b463-daca81f664eb","Type":"ContainerStarted","Data":"73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0"} Nov 02 18:31:24.934311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:24.934266 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:31:24.934537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:24.934475 5647 patch_prober.go:29] interesting pod/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Nov 02 18:31:24.934590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:24.934532 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Nov 02 18:31:25.934300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:25.934260 5647 patch_prober.go:29] interesting pod/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Nov 02 18:31:25.934861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:25.934337 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Nov 02 18:31:26.934004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:26.933963 5647 patch_prober.go:29] interesting pod/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Nov 02 18:31:26.934216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:26.934033 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Nov 02 18:31:27.933920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:27.933876 5647 patch_prober.go:29] interesting pod/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Nov 02 18:31:27.934341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:27.933945 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Nov 02 18:31:28.934018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:28.933976 5647 patch_prober.go:29] interesting pod/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body= Nov 02 18:31:28.934550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:28.934045 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" Nov 02 18:31:29.969492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:29.969452 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:31:29.970304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:29.970161 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:31:29.971115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:31:29.971096 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:31:35.949049 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:31:35.949006 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:32:04.456872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:04.456826 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:32:05.954832 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:05.954784 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:32:10.691001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.690958 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk"] Nov 02 18:32:10.691847 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691825 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:32:10.691847 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691847 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691865 5647 cpu_manager.go:414] "containerMap: removing container" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691872 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f54a78-6398-47bc-86af-d14b50459c61" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691899 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691907 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691929 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691936 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de5b946-4dfc-43a8-a160-afd9a4e19d77" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.691998 5647 cpu_manager.go:414] "containerMap: removing container" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:32:10.692045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.692006 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e3f5f4-f851-43cb-a130-be00f8b00552" containerName="cleanup" Nov 02 18:32:10.692457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.692173 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecaec1de-cb74-495d-b2bb-510d9bb9624d" containerName="cleanup" Nov 02 18:32:10.692457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.692195 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4a3b6d-cbc6-4cc1-b8aa-3dc1f9f28970" containerName="cleanup" Nov 02 18:32:10.692457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.692207 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdac9796-0253-43bd-bb06-1f10bf38f253" containerName="cleanup" Nov 02 18:32:10.695985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.695963 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.752900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.752868 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk"] Nov 02 18:32:10.798947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.798915 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a51e27-serving-cert\"" Nov 02 18:32:10.800046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.800021 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.800162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.800146 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.800322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.800303 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.800392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.800331 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqp5k\" (UniqueName: \"kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.927459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.927420 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.927659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.927589 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.928001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.927897 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.928128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.928019 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqp5k\" (UniqueName: \"kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.928128 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:10.927978 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-serving-cert: secret "maas-api-e2e-inf-b-a51e27-serving-cert" not found Nov 02 18:32:10.928128 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:10.928120 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls podName:400d7ef0-ba41-4e7f-9ea9-ed886038ec0d nodeName:}" failed. No retries permitted until 2026-11-02 18:32:11.428103232 +0000 UTC m=+4118.501630414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls") pod "maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" (UID: "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d") : secret "maas-api-e2e-inf-b-a51e27-serving-cert" not found Nov 02 18:32:10.928383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.928361 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:10.933257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:10.933233 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:11.026548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:11.026463 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqp5k\" (UniqueName: \"kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:11.441061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:11.441028 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:11.443594 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:11.443563 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:11.634934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:11.634898 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:12.014822 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:12.014796 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk"] Nov 02 18:32:12.016043 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:32:12.016018 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400d7ef0_ba41_4e7f_9ea9_ed886038ec0d.slice/crio-8157933fbf17539fb91229fb574df17c686df08472836bd925ccad32ad1a9a9d WatchSource:0}: Error finding container 8157933fbf17539fb91229fb574df17c686df08472836bd925ccad32ad1a9a9d: Status 404 returned error can't find the container with id 8157933fbf17539fb91229fb574df17c686df08472836bd925ccad32ad1a9a9d Nov 02 18:32:12.846606 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:12.846567 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" event={"ID":"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d","Type":"ContainerStarted","Data":"8157933fbf17539fb91229fb574df17c686df08472836bd925ccad32ad1a9a9d"} Nov 02 18:32:13.855329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:13.855287 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" event={"ID":"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d","Type":"ContainerStarted","Data":"b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0"} Nov 02 18:32:13.855806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:13.855456 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:13.971670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:13.971602 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" podStartSLOduration=3.132926569 podStartE2EDuration="3.971583307s" totalImagesPullingTime="838.656738ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:32:13.97036952 +0000 UTC m=+4121.043896760" watchObservedRunningTime="2026-11-02 18:32:13.971583307 +0000 UTC m=+4121.045110514" Nov 02 18:32:15.914379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:15.914340 5647 generic.go:356] "Generic (PLEG): container finished" podID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerID="95f30523fabfcd5e091156060ed9a012b62b6dd75db0f89539b7f94832079f72" exitCode=0 Nov 02 18:32:15.915115 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:15.914692 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" event={"ID":"6afb4a63-964c-4903-8372-7f6cdd4022ff","Type":"ContainerDied","Data":"95f30523fabfcd5e091156060ed9a012b62b6dd75db0f89539b7f94832079f72"} Nov 02 18:32:15.915715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:15.915694 5647 scope.go:122] "RemoveContainer" containerID="95f30523fabfcd5e091156060ed9a012b62b6dd75db0f89539b7f94832079f72" Nov 02 18:32:15.918398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:15.918359 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:32:15.918502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:15.918481 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:32:15.918562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:15.918545 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:32:16.936455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:16.936412 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" event={"ID":"6afb4a63-964c-4903-8372-7f6cdd4022ff","Type":"ContainerStarted","Data":"69296e1f79dbec778936523e0e34c18a1596296bf832db7a3778e339f0b422db"} Nov 02 18:32:17.026762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:17.024906 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:32:17.026762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:17.025396 5647 patch_prober.go:29] interesting pod/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 02 18:32:17.026762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:17.025461 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 02 18:32:18.027825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:18.027255 5647 patch_prober.go:29] interesting pod/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 02 18:32:18.027825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:18.027323 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 02 18:32:19.027288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:19.027252 5647 patch_prober.go:29] interesting pod/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 02 18:32:19.027517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:19.027314 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 02 18:32:19.869432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:19.869403 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:32:20.037617 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:20.037556 5647 patch_prober.go:29] interesting pod/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 02 18:32:20.037836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:20.037645 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 02 18:32:21.026224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:21.026184 5647 patch_prober.go:29] interesting pod/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 02 18:32:21.026885 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:21.026837 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 02 18:32:22.025347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:22.025301 5647 patch_prober.go:29] interesting pod/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" start-of-body= Nov 02 18:32:22.025583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:22.025370 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.21:15021/healthz/ready\": dial tcp 10.134.1.21:15021: connect: connection refused" Nov 02 18:32:23.029074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:23.029040 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:32:23.029575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:23.029424 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:32:23.030146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:23.030128 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:32:30.154823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:30.154786 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp"] Nov 02 18:32:30.261286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:30.261246 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls"] Nov 02 18:32:35.156091 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.156027 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" podUID="f97ba79d-5e3e-44f5-8833-96522d933e16" containerName="payload-processing" containerID="cri-o://7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42" gracePeriod=25 Nov 02 18:32:35.302073 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.302000 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" podUID="d4be0196-409c-4a35-841d-eb59afc69c26" containerName="payload-pre-processing" containerID="cri-o://33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f" gracePeriod=25 Nov 02 18:32:35.514020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.513988 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:32:35.607005 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.606979 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm\" (UniqueName: \"kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm\") pod \"f97ba79d-5e3e-44f5-8833-96522d933e16\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " Nov 02 18:32:35.607161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.607017 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume\") pod \"f97ba79d-5e3e-44f5-8833-96522d933e16\" (UID: \"f97ba79d-5e3e-44f5-8833-96522d933e16\") " Nov 02 18:32:35.607444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.607415 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume" pod "f97ba79d-5e3e-44f5-8833-96522d933e16" (UID: "f97ba79d-5e3e-44f5-8833-96522d933e16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:32:35.607600 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.607548 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:32:35.607680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.607665 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97ba79d-5e3e-44f5-8833-96522d933e16-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:32:35.609335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.609308 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm" pod "f97ba79d-5e3e-44f5-8833-96522d933e16" (UID: "f97ba79d-5e3e-44f5-8833-96522d933e16"). InnerVolumeSpecName "kube-api-access-p8bvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:35.708825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.708726 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44\" (UniqueName: \"kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44\") pod \"d4be0196-409c-4a35-841d-eb59afc69c26\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " Nov 02 18:32:35.709017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.708915 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume\") pod \"d4be0196-409c-4a35-841d-eb59afc69c26\" (UID: \"d4be0196-409c-4a35-841d-eb59afc69c26\") " Nov 02 18:32:35.709274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.709245 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume" pod "d4be0196-409c-4a35-841d-eb59afc69c26" (UID: "d4be0196-409c-4a35-841d-eb59afc69c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:32:35.709458 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.709438 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8bvm\" (UniqueName: \"kubernetes.io/projected/f97ba79d-5e3e-44f5-8833-96522d933e16-kube-api-access-p8bvm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:32:35.709552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.709458 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4be0196-409c-4a35-841d-eb59afc69c26-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:32:35.711261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.711229 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44" pod "d4be0196-409c-4a35-841d-eb59afc69c26" (UID: "d4be0196-409c-4a35-841d-eb59afc69c26"). InnerVolumeSpecName "kube-api-access-ctj44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:32:35.810436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:35.810392 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctj44\" (UniqueName: \"kubernetes.io/projected/d4be0196-409c-4a35-841d-eb59afc69c26-kube-api-access-ctj44\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:32:35.950148 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:35.950104 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:32:36.098818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.098782 5647 generic.go:356] "Generic (PLEG): container finished" podID="f97ba79d-5e3e-44f5-8833-96522d933e16" containerID="7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42" exitCode=0 Nov 02 18:32:36.099030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.098851 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" Nov 02 18:32:36.099030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.098873 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" event={"ID":"f97ba79d-5e3e-44f5-8833-96522d933e16","Type":"ContainerDied","Data":"7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42"} Nov 02 18:32:36.099030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.098939 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp" event={"ID":"f97ba79d-5e3e-44f5-8833-96522d933e16","Type":"ContainerDied","Data":"de0d751784688d76f87454e2ed101eb74f923f89f83a0eb63ac9a4a26fc642c1"} Nov 02 18:32:36.099030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.098969 5647 scope.go:122] "RemoveContainer" containerID="7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42" Nov 02 18:32:36.100473 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.100448 5647 generic.go:356] "Generic (PLEG): container finished" podID="d4be0196-409c-4a35-841d-eb59afc69c26" containerID="33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f" exitCode=0 Nov 02 18:32:36.100592 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.100487 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" Nov 02 18:32:36.100686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.100664 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" event={"ID":"d4be0196-409c-4a35-841d-eb59afc69c26","Type":"ContainerDied","Data":"33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f"} Nov 02 18:32:36.100796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.100716 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls" event={"ID":"d4be0196-409c-4a35-841d-eb59afc69c26","Type":"ContainerDied","Data":"e0b0e2820934e52abd8edc3b737575e4ac58ea53cf7dbb7d33190969972f683f"} Nov 02 18:32:36.110021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.110001 5647 scope.go:122] "RemoveContainer" containerID="7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42" Nov 02 18:32:36.110350 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:36.110329 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42\": container with ID starting with 7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42 not found: ID does not exist" containerID="7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42" Nov 02 18:32:36.110411 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.110360 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42"} err="failed to get container status \"7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42\": rpc error: code = NotFound desc = could not find container \"7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42\": container with ID starting with 7f97b6c27f5bed8408569bb978b19a1679b6d778377fc5acfaf2dbd1b200df42 not found: ID does not exist" Nov 02 18:32:36.110411 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.110379 5647 scope.go:122] "RemoveContainer" containerID="33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f" Nov 02 18:32:36.119990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.119971 5647 scope.go:122] "RemoveContainer" containerID="33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f" Nov 02 18:32:36.120318 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:36.120296 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f\": container with ID starting with 33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f not found: ID does not exist" containerID="33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f" Nov 02 18:32:36.120371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.120326 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f"} err="failed to get container status \"33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f\": rpc error: code = NotFound desc = could not find container \"33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f\": container with ID starting with 33c1c42409a4d7b62205be13b15538388996cd41f13bd97b4d0ac4fe3094cf7f not found: ID does not exist" Nov 02 18:32:36.265647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.265608 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls"] Nov 02 18:32:36.271969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.271935 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c70712-74ccc5bd9f-tzlls"] Nov 02 18:32:36.383769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.383661 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp"] Nov 02 18:32:36.388156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:36.388125 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c70712-5bbb775d69-ndhhp"] Nov 02 18:32:37.082248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:37.082213 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4be0196-409c-4a35-841d-eb59afc69c26" path="/var/lib/kubelet/pods/d4be0196-409c-4a35-841d-eb59afc69c26/volumes" Nov 02 18:32:37.082694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:37.082679 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97ba79d-5e3e-44f5-8833-96522d933e16" path="/var/lib/kubelet/pods/f97ba79d-5e3e-44f5-8833-96522d933e16/volumes" Nov 02 18:32:37.206337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:37.206304 5647 scope.go:122] "RemoveContainer" containerID="a6c8379bfa4fc90bc556bff4357ecb4c5b47a30ffdb1aeba58e71c9490ffd53d" Nov 02 18:32:37.223842 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:37.223723 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": read tcp 10.134.0.2:48190->10.134.1.22:15021: read: connection reset by peer" start-of-body= Nov 02 18:32:37.223842 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:37.223794 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": read tcp 10.134.0.2:48190->10.134.1.22:15021: read: connection reset by peer" Nov 02 18:32:38.116582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:38.116546 5647 generic.go:356] "Generic (PLEG): container finished" podID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerID="5e5291c127e6cdfc5222440d52ab87e9c5c02cb256dd24cb47d08f8c49688d65" exitCode=0 Nov 02 18:32:38.117016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:38.116621 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" event={"ID":"f393f960-71c5-4ca2-bd2b-d7d1d49b9374","Type":"ContainerDied","Data":"5e5291c127e6cdfc5222440d52ab87e9c5c02cb256dd24cb47d08f8c49688d65"} Nov 02 18:32:38.117097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:38.117081 5647 scope.go:122] "RemoveContainer" containerID="5e5291c127e6cdfc5222440d52ab87e9c5c02cb256dd24cb47d08f8c49688d65" Nov 02 18:32:38.118887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:38.118855 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:32:38.118996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:38.118932 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:32:38.118996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:38.118971 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:32:39.135306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:39.135263 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" event={"ID":"f393f960-71c5-4ca2-bd2b-d7d1d49b9374","Type":"ContainerStarted","Data":"ab9b0db6178e7b7b78ca3e6af1cef3cef66ebfdb0b6cd3fa142026b3fb26fa51"} Nov 02 18:32:39.250507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:39.250468 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:32:39.250697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:39.250675 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:32:39.250789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:39.250721 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:32:40.250663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:40.250627 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:32:40.251116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:40.250694 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:32:41.250442 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:41.250405 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:32:41.250633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:41.250473 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:32:42.250607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:42.250572 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:32:42.251143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:42.250639 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:32:43.250157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:43.250125 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:32:43.250342 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:43.250175 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:32:44.250146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:44.250103 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 02 18:32:44.250671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:44.250189 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 02 18:32:45.254001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:45.253968 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:32:45.254508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:45.254282 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:32:45.255153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:45.255134 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:32:49.093278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.093247 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2"] Nov 02 18:32:49.093864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.093850 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d4be0196-409c-4a35-841d-eb59afc69c26" containerName="payload-pre-processing" Nov 02 18:32:49.093923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.093865 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4be0196-409c-4a35-841d-eb59afc69c26" containerName="payload-pre-processing" Nov 02 18:32:49.093923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.093882 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f97ba79d-5e3e-44f5-8833-96522d933e16" containerName="payload-processing" Nov 02 18:32:49.093923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.093887 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97ba79d-5e3e-44f5-8833-96522d933e16" containerName="payload-processing" Nov 02 18:32:49.094027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.093987 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97ba79d-5e3e-44f5-8833-96522d933e16" containerName="payload-processing" Nov 02 18:32:49.094027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.094003 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4be0196-409c-4a35-841d-eb59afc69c26" containerName="payload-pre-processing" Nov 02 18:32:49.096637 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.096616 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.119436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.119402 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246m9\" (UniqueName: \"kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.119666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.119644 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.119831 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.119803 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.119918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.119864 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.153457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.153424 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2"] Nov 02 18:32:49.200338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.200306 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-608907-serving-cert\"" Nov 02 18:32:49.221083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.221047 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.221083 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.221081 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.221332 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.221134 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246m9\" (UniqueName: \"kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.221332 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:49.221221 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-serving-cert: secret "maas-api-e2e-inf-b-608907-serving-cert" not found Nov 02 18:32:49.221332 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:32:49.221306 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls podName:a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9 nodeName:}" failed. No retries permitted until 2026-11-02 18:32:49.721278258 +0000 UTC m=+4156.794805438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls") pod "maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" (UID: "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9") : secret "maas-api-e2e-inf-b-608907-serving-cert" not found Nov 02 18:32:49.221332 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.221229 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.221893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.221877 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.224453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.224425 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.279267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.279235 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-246m9\" (UniqueName: \"kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.733940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.733903 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:49.736494 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:49.736473 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\") pod \"maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:50.011110 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:50.011024 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:50.387415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:50.387375 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2"] Nov 02 18:32:50.389266 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:32:50.389229 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76c6de9_c11f_43a6_9d9f_baa3cb01d7d9.slice/crio-a917a39ac5b39887b1166640709aaf95365ecb017723a3600285fe48c6b06a2e WatchSource:0}: Error finding container a917a39ac5b39887b1166640709aaf95365ecb017723a3600285fe48c6b06a2e: Status 404 returned error can't find the container with id a917a39ac5b39887b1166640709aaf95365ecb017723a3600285fe48c6b06a2e Nov 02 18:32:51.234793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:51.234724 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" event={"ID":"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9","Type":"ContainerStarted","Data":"a917a39ac5b39887b1166640709aaf95365ecb017723a3600285fe48c6b06a2e"} Nov 02 18:32:52.243839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:52.243807 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" event={"ID":"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9","Type":"ContainerStarted","Data":"822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0"} Nov 02 18:32:52.244277 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:52.243947 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:32:52.363391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:52.363336 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" podStartSLOduration=2.466910629 podStartE2EDuration="3.363321021s" totalImagesPullingTime="896.410392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:32:52.359600909 +0000 UTC m=+4159.433128123" watchObservedRunningTime="2026-11-02 18:32:52.363321021 +0000 UTC m=+4159.436848223" Nov 02 18:32:58.270053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:32:58.270024 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:33:03.718762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:03.718029 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:33:05.948871 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:33:05.948823 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:33:22.112456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:22.112417 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq"] Nov 02 18:33:27.114435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.114396 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" podUID="515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" containerName="payload-processing" containerID="cri-o://14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5" gracePeriod=25 Nov 02 18:33:27.428026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.427998 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:33:27.501965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.501928 5647 generic.go:356] "Generic (PLEG): container finished" podID="515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" containerID="14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5" exitCode=0 Nov 02 18:33:27.502194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.501990 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" Nov 02 18:33:27.502194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.502020 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" event={"ID":"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d","Type":"ContainerDied","Data":"14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5"} Nov 02 18:33:27.502194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.502075 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq" event={"ID":"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d","Type":"ContainerDied","Data":"c88988b771a27ad123a7d4214bff8853a52ced446858c43f911fbb12c6de62b8"} Nov 02 18:33:27.502194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.502103 5647 scope.go:122] "RemoveContainer" containerID="14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5" Nov 02 18:33:27.512759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.512705 5647 scope.go:122] "RemoveContainer" containerID="14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5" Nov 02 18:33:27.513054 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:33:27.513031 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5\": container with ID starting with 14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5 not found: ID does not exist" containerID="14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5" Nov 02 18:33:27.513112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.513063 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5"} err="failed to get container status \"14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5\": rpc error: code = NotFound desc = could not find container \"14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5\": container with ID starting with 14179b21e0126ad26a8705fb7c1a8af4a0fbca0fa2ac107ac3f04c59b28ebcb5 not found: ID does not exist" Nov 02 18:33:27.518561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.518537 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume\") pod \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " Nov 02 18:33:27.518876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.518861 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2\" (UniqueName: \"kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2\") pod \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\" (UID: \"515f836f-91bf-45aa-a4a9-e8eaed8a1b0d\") " Nov 02 18:33:27.518932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.518908 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume" pod "515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" (UID: "515f836f-91bf-45aa-a4a9-e8eaed8a1b0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:33:27.519241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.519227 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.521109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.521084 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2" pod "515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" (UID: "515f836f-91bf-45aa-a4a9-e8eaed8a1b0d"). InnerVolumeSpecName "kube-api-access-rg6v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:33:27.620503 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.620464 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg6v2\" (UniqueName: \"kubernetes.io/projected/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d-kube-api-access-rg6v2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:33:27.933878 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.933838 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq"] Nov 02 18:33:27.936634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:27.936607 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-662a80-7f9bc74b47-r5nbq"] Nov 02 18:33:29.082056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:29.082018 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" path="/var/lib/kubelet/pods/515f836f-91bf-45aa-a4a9-e8eaed8a1b0d/volumes" Nov 02 18:33:34.561318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:34.561272 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:33:34.570830 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:34.570805 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:33:35.949043 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:33:35.949007 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:33:58.945536 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:58.945490 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96"] Nov 02 18:33:58.946031 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:58.945831 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" containerID="cri-o://fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4" gracePeriod=30 Nov 02 18:33:58.994468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:58.994429 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz"] Nov 02 18:33:58.994674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:58.994643 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" containerID="cri-o://b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9" gracePeriod=30 Nov 02 18:33:58.996966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:58.996934 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst"] Nov 02 18:33:58.997156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:58.997121 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" containerID="cri-o://8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1" gracePeriod=30 Nov 02 18:33:59.045460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:59.045429 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg"] Nov 02 18:33:59.045642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:33:59.045623 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" containerID="cri-o://ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa" gracePeriod=30 Nov 02 18:34:02.499766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:02.499715 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-767tz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:34:02.500183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:02.499798 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:34:04.302936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.302902 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:34:04.360663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.360634 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:34:04.415193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.415162 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:34:04.419433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.419404 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 18:34:04.432721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.432695 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.432888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.432781 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.432888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.432834 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.432979 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.433077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433021 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433117 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433152 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433183 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433222 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433253 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.433325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433280 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433322 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433359 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433393 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433416 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.433496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433480 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433528 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433576 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433611 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.433799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433649 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert\") pod \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\" (UID: \"3f84129a-bb35-4c76-9db8-bb15ea5fcd36\") " Nov 02 18:34:04.433799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433757 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk\") pod \"b0ef0310-95fb-48c0-82d0-8008c5131787\" (UID: \"b0ef0310-95fb-48c0-82d0-8008c5131787\") " Nov 02 18:34:04.434061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.433848 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434304 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434479 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434503 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434512 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434518 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434553 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0ef0310-95fb-48c0-82d0-8008c5131787-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434572 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434844 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:04.434948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434799 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.436069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.434978 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.436069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.435061 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.436069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.435909 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.436069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.436019 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:04.436069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.436045 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.436506 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.436487 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "kube-api-access-rtgpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.436619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.436595 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.436728 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.436688 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:04.437630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.437613 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token" pod "3f84129a-bb35-4c76-9db8-bb15ea5fcd36" (UID: "3f84129a-bb35-4c76-9db8-bb15ea5fcd36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.437856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.437835 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk" pod "b0ef0310-95fb-48c0-82d0-8008c5131787" (UID: "b0ef0310-95fb-48c0-82d0-8008c5131787"). InnerVolumeSpecName "kube-api-access-fmmjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.535418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535335 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.535418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535417 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535452 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535480 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535503 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535531 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535592 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535619 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.535680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535666 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535656 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535694 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535783 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535801 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535810 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535867 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535901 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535911 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.535990 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.536034 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.536112 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert\") pod \"4d78f16e-044f-4b75-aeff-3eec744367ad\" (UID: \"4d78f16e-044f-4b75-aeff-3eec744367ad\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.536147 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.536152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.536181 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket\") pod \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\" (UID: \"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45\") " Nov 02 18:34:04.537089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537060 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmmjk\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-kube-api-access-fmmjk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537085 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537100 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537117 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537133 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537149 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537163 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtgpw\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-kube-api-access-rtgpw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537179 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537193 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537209 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0ef0310-95fb-48c0-82d0-8008c5131787-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537226 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537240 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537254 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537271 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537285 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0ef0310-95fb-48c0-82d0-8008c5131787-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537300 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537314 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84129a-bb35-4c76-9db8-bb15ea5fcd36-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.537661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537556 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.538134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.537941 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.538198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.538141 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.538346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.538310 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:04.538434 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.538388 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:34:04.538588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.538556 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.538705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.538681 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.538992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.538877 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:34:04.539155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.539017 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "kube-api-access-mdtwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.539155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.539056 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.539375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.539161 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.539375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.539220 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:04.540207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.540186 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct" pod "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" (UID: "377b6b6a-3a4d-4b68-8b8b-bd40629ddf45"). InnerVolumeSpecName "kube-api-access-dglct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:34:04.540567 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.540550 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo" pod "4d78f16e-044f-4b75-aeff-3eec744367ad" (UID: "4d78f16e-044f-4b75-aeff-3eec744367ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:34:04.638651 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638609 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638651 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638647 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638661 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638673 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638682 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdtwc\" (UniqueName: \"kubernetes.io/projected/4d78f16e-044f-4b75-aeff-3eec744367ad-kube-api-access-mdtwc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638692 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638702 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglct\" (UniqueName: \"kubernetes.io/projected/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-kube-api-access-dglct\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638711 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638722 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638752 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d78f16e-044f-4b75-aeff-3eec744367ad-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638761 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638769 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638778 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.638920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.638786 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d78f16e-044f-4b75-aeff-3eec744367ad-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:34:04.802501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.802413 5647 generic.go:356] "Generic (PLEG): container finished" podID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerID="ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa" exitCode=0 Nov 02 18:34:04.802501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.802466 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" Nov 02 18:34:04.802768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.802491 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" event={"ID":"b0ef0310-95fb-48c0-82d0-8008c5131787","Type":"ContainerDied","Data":"ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa"} Nov 02 18:34:04.802768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.802556 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg" event={"ID":"b0ef0310-95fb-48c0-82d0-8008c5131787","Type":"ContainerDied","Data":"c09b8d007b4762eb7f09000c3c79d79ff7ebb640875b900394b74a0055717c09"} Nov 02 18:34:04.802768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.802582 5647 scope.go:122] "RemoveContainer" containerID="ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa" Nov 02 18:34:04.804077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.804051 5647 generic.go:356] "Generic (PLEG): container finished" podID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerID="fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4" exitCode=0 Nov 02 18:34:04.804230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.804090 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" Nov 02 18:34:04.804230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.804128 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" event={"ID":"3f84129a-bb35-4c76-9db8-bb15ea5fcd36","Type":"ContainerDied","Data":"fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4"} Nov 02 18:34:04.804230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.804187 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96" event={"ID":"3f84129a-bb35-4c76-9db8-bb15ea5fcd36","Type":"ContainerDied","Data":"7ad8aa3e3ab54f2971b49fe26be5d90ca254c9e8263c51fccf3f3ed41e481ac9"} Nov 02 18:34:04.805829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.805793 5647 generic.go:356] "Generic (PLEG): container finished" podID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerID="b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9" exitCode=0 Nov 02 18:34:04.805946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.805832 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" Nov 02 18:34:04.805946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.805868 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" event={"ID":"4d78f16e-044f-4b75-aeff-3eec744367ad","Type":"ContainerDied","Data":"b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9"} Nov 02 18:34:04.805946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.805913 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz" event={"ID":"4d78f16e-044f-4b75-aeff-3eec744367ad","Type":"ContainerDied","Data":"6e2b2bda773f74743f896b4aa20f69a31e1fc0ef534b4d692318974899ce5e4b"} Nov 02 18:34:04.808003 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.807977 5647 generic.go:356] "Generic (PLEG): container finished" podID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerID="8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1" exitCode=0 Nov 02 18:34:04.808127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.808019 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" event={"ID":"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45","Type":"ContainerDied","Data":"8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1"} Nov 02 18:34:04.808127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.808027 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" Nov 02 18:34:04.808127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.808055 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst" event={"ID":"377b6b6a-3a4d-4b68-8b8b-bd40629ddf45","Type":"ContainerDied","Data":"267047d2d74e165911df9a609931b021ff02e84a34f8d2cfe7ef174bec63ec76"} Nov 02 18:34:04.818038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.817924 5647 scope.go:122] "RemoveContainer" containerID="ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa" Nov 02 18:34:04.818609 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:34:04.818588 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa\": container with ID starting with ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa not found: ID does not exist" containerID="ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa" Nov 02 18:34:04.818686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.818618 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa"} err="failed to get container status \"ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa\": rpc error: code = NotFound desc = could not find container \"ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa\": container with ID starting with ac6b92c59bf38551e9bc838998dd55e597f74fbd7208c7ac1e96421b10395caa not found: ID does not exist" Nov 02 18:34:04.818686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.818638 5647 scope.go:122] "RemoveContainer" containerID="fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4" Nov 02 18:34:04.829801 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.829778 5647 scope.go:122] "RemoveContainer" containerID="fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4" Nov 02 18:34:04.830388 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:34:04.830350 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4\": container with ID starting with fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4 not found: ID does not exist" containerID="fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4" Nov 02 18:34:04.830670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.830538 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4"} err="failed to get container status \"fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4\": rpc error: code = NotFound desc = could not find container \"fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4\": container with ID starting with fdb40968eeb520a808e07a06a3826648e6053373d5ae2729af841f9e98a8adc4 not found: ID does not exist" Nov 02 18:34:04.830670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.830569 5647 scope.go:122] "RemoveContainer" containerID="b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9" Nov 02 18:34:04.961964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.961839 5647 scope.go:122] "RemoveContainer" containerID="b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9" Nov 02 18:34:04.982969 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:34:04.962271 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9\": container with ID starting with b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9 not found: ID does not exist" containerID="b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9" Nov 02 18:34:04.982969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.962325 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9"} err="failed to get container status \"b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9\": rpc error: code = NotFound desc = could not find container \"b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9\": container with ID starting with b5018254eff4363050f5def1dbc58666580d16a68d0bed0ee2d7fcf98d2f9eb9 not found: ID does not exist" Nov 02 18:34:04.982969 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.962349 5647 scope.go:122] "RemoveContainer" containerID="8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1" Nov 02 18:34:04.987601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.987566 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg"] Nov 02 18:34:04.993231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:04.993194 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-jwkvg"] Nov 02 18:34:05.015931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.015910 5647 scope.go:122] "RemoveContainer" containerID="8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1" Nov 02 18:34:05.016291 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:34:05.016268 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1\": container with ID starting with 8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1 not found: ID does not exist" containerID="8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1" Nov 02 18:34:05.016336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.016306 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1"} err="failed to get container status \"8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1\": rpc error: code = NotFound desc = could not find container \"8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1\": container with ID starting with 8d671bc110ddb8d2e77484cd74b39befe6f808e311cafe7c75707d58269d94e1 not found: ID does not exist" Nov 02 18:34:05.081562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.081487 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" path="/var/lib/kubelet/pods/b0ef0310-95fb-48c0-82d0-8008c5131787/volumes" Nov 02 18:34:05.106723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.106571 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96"] Nov 02 18:34:05.109541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.109514 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-s4p96"] Nov 02 18:34:05.220639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.220581 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst"] Nov 02 18:34:05.226155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.226130 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-g8qst"] Nov 02 18:34:05.266884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.266851 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq"] Nov 02 18:34:05.267481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267463 5647 cpu_manager.go:414] "containerMap: removing container" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" Nov 02 18:34:05.267481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267482 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267512 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267518 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267537 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267542 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267553 5647 cpu_manager.go:414] "containerMap: removing container" podUID="515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" containerName="payload-processing" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267558 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" containerName="payload-processing" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267569 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" Nov 02 18:34:05.267622 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267574 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" Nov 02 18:34:05.267927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267650 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" containerName="istio-proxy" Nov 02 18:34:05.267927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267661 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" containerName="istio-proxy" Nov 02 18:34:05.267927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267669 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="515f836f-91bf-45aa-a4a9-e8eaed8a1b0d" containerName="payload-processing" Nov 02 18:34:05.267927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267675 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ef0310-95fb-48c0-82d0-8008c5131787" containerName="istio-proxy" Nov 02 18:34:05.267927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.267684 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" containerName="istio-proxy" Nov 02 18:34:05.272822 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.272803 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.326452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.326411 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-134555-openshift-default-dockercfg-44tk7\"" Nov 02 18:34:05.329314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.329286 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq"] Nov 02 18:34:05.339245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.339214 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz"] Nov 02 18:34:05.342788 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.342762 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-767tz"] Nov 02 18:34:05.362548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.362520 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.362702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.362571 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.362779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.362726 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.362852 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.362837 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.362901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.362885 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.363001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.362950 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.363052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.363019 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.363052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.363043 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.363176 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.363152 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kkz\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464271 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464326 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464380 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464417 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464450 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464491 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.464539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464536 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464564 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464601 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kkz\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.464929 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.465051 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465342 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.465187 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465342 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.465214 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.465342 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.465334 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.467014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.466980 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.467145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.467116 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.548078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.548040 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.551030 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.550603 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kkz\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz\") pod \"e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.586695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:05.586664 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:05.947409 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:34:05.947319 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:34:06.015191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.015153 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq"] Nov 02 18:34:06.017070 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:34:06.017039 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75529ed_454c_417f_8bc5_4b6bf3840a72.slice/crio-8766d4970615bb948d556de42cfd8887bac030aaecd4fbf044ddd42cda032ccf WatchSource:0}: Error finding container 8766d4970615bb948d556de42cfd8887bac030aaecd4fbf044ddd42cda032ccf: Status 404 returned error can't find the container with id 8766d4970615bb948d556de42cfd8887bac030aaecd4fbf044ddd42cda032ccf Nov 02 18:34:06.020325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.020283 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:06.020410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.020382 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:06.020461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.020426 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:06.841552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.841512 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" event={"ID":"b75529ed-454c-417f-8bc5-4b6bf3840a72","Type":"ContainerStarted","Data":"7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226"} Nov 02 18:34:06.842025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.841595 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" event={"ID":"b75529ed-454c-417f-8bc5-4b6bf3840a72","Type":"ContainerStarted","Data":"8766d4970615bb948d556de42cfd8887bac030aaecd4fbf044ddd42cda032ccf"} Nov 02 18:34:06.960782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:06.960703 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podStartSLOduration=1.960685782 podStartE2EDuration="1.960685782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:06.958411453 +0000 UTC m=+4234.031938659" watchObservedRunningTime="2026-11-02 18:34:06.960685782 +0000 UTC m=+4234.034212984" Nov 02 18:34:07.125062 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.124941 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="377b6b6a-3a4d-4b68-8b8b-bd40629ddf45" path="/var/lib/kubelet/pods/377b6b6a-3a4d-4b68-8b8b-bd40629ddf45/volumes" Nov 02 18:34:07.126105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.126042 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f84129a-bb35-4c76-9db8-bb15ea5fcd36" path="/var/lib/kubelet/pods/3f84129a-bb35-4c76-9db8-bb15ea5fcd36/volumes" Nov 02 18:34:07.126990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.126971 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d78f16e-044f-4b75-aeff-3eec744367ad" path="/var/lib/kubelet/pods/4d78f16e-044f-4b75-aeff-3eec744367ad/volumes" Nov 02 18:34:07.587559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.587533 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:07.593962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.593936 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:07.848310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.848227 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:07.849345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:07.849325 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:34:13.665837 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.665682 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb"] Nov 02 18:34:13.765122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.765086 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb"] Nov 02 18:34:13.765334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.765220 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:13.801910 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.801880 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:13.802113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.802092 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srhm\" (UniqueName: \"kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm\") pod \"payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:13.819517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.819480 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-134555\"" Nov 02 18:34:13.870117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.870077 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-134555-dockercfg-7p69m\"" Nov 02 18:34:13.903255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.903214 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:13.903470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.903398 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srhm\" (UniqueName: \"kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm\") pod \"payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:13.903991 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.903966 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:13.960987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:13.960914 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srhm\" (UniqueName: \"kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm\") pod \"payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:14.082063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:14.082021 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:14.463262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:14.463234 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb"] Nov 02 18:34:14.464633 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:34:14.464600 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784b5e8a_a63d_4d21_8a57_8d2f29d095f3.slice/crio-9c508fc86b0b8a204f85905ab12e1c3a5f2ca11d952e91b7d6bb6b3c552a4123 WatchSource:0}: Error finding container 9c508fc86b0b8a204f85905ab12e1c3a5f2ca11d952e91b7d6bb6b3c552a4123: Status 404 returned error can't find the container with id 9c508fc86b0b8a204f85905ab12e1c3a5f2ca11d952e91b7d6bb6b3c552a4123 Nov 02 18:34:14.895693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:14.895652 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" event={"ID":"784b5e8a-a63d-4d21-8a57-8d2f29d095f3","Type":"ContainerStarted","Data":"17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5"} Nov 02 18:34:14.896178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:14.895721 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" event={"ID":"784b5e8a-a63d-4d21-8a57-8d2f29d095f3","Type":"ContainerStarted","Data":"9c508fc86b0b8a204f85905ab12e1c3a5f2ca11d952e91b7d6bb6b3c552a4123"} Nov 02 18:34:14.896178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:14.895886 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:15.013491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:15.013434 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" podStartSLOduration=2.013419968 podStartE2EDuration="2.013419968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:15.010865584 +0000 UTC m=+4242.084392788" watchObservedRunningTime="2026-11-02 18:34:15.013419968 +0000 UTC m=+4242.086947171" Nov 02 18:34:18.905056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:18.905016 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:34:25.906159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:25.906129 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:34:35.949264 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:34:35.949221 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:34:44.318048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.318012 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb"] Nov 02 18:34:44.323052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.323012 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.368819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.368788 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc"] Nov 02 18:34:44.372899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.372876 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb"] Nov 02 18:34:44.373020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.372999 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.376849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.376827 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.412230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412190 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qft7z\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.412423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412243 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.412423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412322 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.412423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412363 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.412423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412384 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.412423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412418 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.412671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412514 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf6dg\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.412671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412541 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.412671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412562 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.412876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412672 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.412876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412709 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.412876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412804 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.412876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412835 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.412876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412862 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.413093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412898 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.413093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.412977 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.413093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413003 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzmz\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.413265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413197 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.413265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413228 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.413361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413262 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.413361 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413324 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.413454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413439 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.413679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413654 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.413829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413700 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.413829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413727 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.413970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413947 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.414025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.413989 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.416839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.416818 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\"" Nov 02 18:34:44.421168 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.421145 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb"] Nov 02 18:34:44.476966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.476928 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc"] Nov 02 18:34:44.484132 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.484094 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb"] Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.523260 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.526907 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.526951 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.526984 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527041 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527072 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzmz\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527088 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527297 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527328 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527362 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527386 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527427 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527494 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527529 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527554 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527599 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527639 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527681 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qft7z\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527750 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527781 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527811 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527839 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527891 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527944 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf6dg\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.527989 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.528027 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.528085 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.528111 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.529076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.528614 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.531063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.529885 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.531063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.530149 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.531063 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.530398 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.531304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.531190 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.536222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.531760 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.536222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.534852 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.536222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.535103 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.537884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.536644 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.537884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.537149 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.537884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.537507 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.537884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.537811 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.538522 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.538484 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.538806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.538773 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.546987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.546959 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.547783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.547686 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.549029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.548914 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.549029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.548983 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.549603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.549581 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.549717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.549700 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.585786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.585673 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzmz\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.590129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.590103 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qft7z\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.624626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.624598 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-7bckb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:44.624839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.624689 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-xqvvb\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.624921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.624856 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.624921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.624909 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf6dg\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-8flxc\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.640327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.640293 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:44.688866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.688833 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:44.695972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:44.694680 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:45.070992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.070958 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb"] Nov 02 18:34:45.110778 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:34:45.110747 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342fc860_018d_4d7b_9838_4578ab8e1aa9.slice/crio-63f63d257e9a7b32d67d0e9dc97528e66591f7638d79fa757ad6c0e99213488f WatchSource:0}: Error finding container 63f63d257e9a7b32d67d0e9dc97528e66591f7638d79fa757ad6c0e99213488f: Status 404 returned error can't find the container with id 63f63d257e9a7b32d67d0e9dc97528e66591f7638d79fa757ad6c0e99213488f Nov 02 18:34:45.113944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.113909 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.114061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.113990 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.114061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.114034 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.123720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.123695 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb"] Nov 02 18:34:45.125598 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:34:45.125574 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca373f8d_d622_429a_944e_5508eae9e3fb.slice/crio-2758bdba4cedc5ec681d6b5c81d9049593f0987200c8ff40daa114b1ca4f0ae7 WatchSource:0}: Error finding container 2758bdba4cedc5ec681d6b5c81d9049593f0987200c8ff40daa114b1ca4f0ae7: Status 404 returned error can't find the container with id 2758bdba4cedc5ec681d6b5c81d9049593f0987200c8ff40daa114b1ca4f0ae7 Nov 02 18:34:45.128450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.128413 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.128572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.128501 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.128572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.128545 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.172994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.172964 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc"] Nov 02 18:34:45.174428 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:34:45.174400 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9decc592_ea53_42b3_9f3f_eeaba805f7a2.slice/crio-a85b8d71a99aa466ac1a2c96e85ce3969372b2ad2f19447b9a148035fc7debb9 WatchSource:0}: Error finding container a85b8d71a99aa466ac1a2c96e85ce3969372b2ad2f19447b9a148035fc7debb9: Status 404 returned error can't find the container with id a85b8d71a99aa466ac1a2c96e85ce3969372b2ad2f19447b9a148035fc7debb9 Nov 02 18:34:45.177384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.177351 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.177448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.177426 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:45.177489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:45.177453 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:34:46.161675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.161611 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" event={"ID":"9decc592-ea53-42b3-9f3f-eeaba805f7a2","Type":"ContainerStarted","Data":"0b76941a7c0fd94c23156d8c9018b479a92fc2d84cc6825ccebf214458578e11"} Nov 02 18:34:46.162238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.161694 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" event={"ID":"9decc592-ea53-42b3-9f3f-eeaba805f7a2","Type":"ContainerStarted","Data":"a85b8d71a99aa466ac1a2c96e85ce3969372b2ad2f19447b9a148035fc7debb9"} Nov 02 18:34:46.174647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.174605 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" event={"ID":"ca373f8d-d622-429a-944e-5508eae9e3fb","Type":"ContainerStarted","Data":"ca296b09f6a8eaf937438a68356d52dedffb8e98a6c189dd9c4110b0956e6e4a"} Nov 02 18:34:46.174968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.174946 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" event={"ID":"ca373f8d-d622-429a-944e-5508eae9e3fb","Type":"ContainerStarted","Data":"2758bdba4cedc5ec681d6b5c81d9049593f0987200c8ff40daa114b1ca4f0ae7"} Nov 02 18:34:46.182758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.182277 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" event={"ID":"342fc860-018d-4d7b-9838-4578ab8e1aa9","Type":"ContainerStarted","Data":"06534d0ce220bbaa02c52f8d17e4bda7998c1bf7ce0e3f092f3302fb4c708737"} Nov 02 18:34:46.182758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.182345 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" event={"ID":"342fc860-018d-4d7b-9838-4578ab8e1aa9","Type":"ContainerStarted","Data":"63f63d257e9a7b32d67d0e9dc97528e66591f7638d79fa757ad6c0e99213488f"} Nov 02 18:34:46.287277 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.287215 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podStartSLOduration=2.287197126 podStartE2EDuration="2.287197126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:46.28623784 +0000 UTC m=+4273.359765046" watchObservedRunningTime="2026-11-02 18:34:46.287197126 +0000 UTC m=+4273.360724332" Nov 02 18:34:46.452832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.432626 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podStartSLOduration=2.43260518 podStartE2EDuration="2.43260518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:46.411290803 +0000 UTC m=+4273.484818019" watchObservedRunningTime="2026-11-02 18:34:46.43260518 +0000 UTC m=+4273.506132383" Nov 02 18:34:46.531757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.531642 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podStartSLOduration=2.531623057 podStartE2EDuration="2.531623057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:46.530687641 +0000 UTC m=+4273.604214845" watchObservedRunningTime="2026-11-02 18:34:46.531623057 +0000 UTC m=+4273.605150261" Nov 02 18:34:46.646758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.641082 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:46.646758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.643397 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:46.646758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.643454 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:46.689995 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.689954 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:46.691294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.691273 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:46.691400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.691324 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:46.695476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.695457 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:46.696410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.696384 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:46.696460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:46.696423 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:47.644881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:47.644830 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:47.645423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:47.644896 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:47.690298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:47.690175 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:47.690298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:47.690238 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:47.696436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:47.696397 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:47.696656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:47.696459 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:48.644757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:48.644118 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:48.644757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:48.644180 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:48.689758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:48.689560 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:48.689758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:48.689609 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:48.695989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:48.695962 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:48.696087 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:48.696007 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:49.644759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:49.644323 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:49.644759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:49.644404 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:49.689925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:49.689884 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:49.690119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:49.689960 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:49.695405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:49.695375 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:49.695542 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:49.695425 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:50.644761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:50.640968 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:50.644761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:50.641048 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:50.689570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:50.689532 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:50.690245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:50.690213 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:50.695922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:50.695896 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:50.696058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:50.695941 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:51.640870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:51.640658 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:51.640870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:51.640725 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:51.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:51.690301 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:51.690789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:51.690374 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:51.696082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:51.696046 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:51.696239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:51.696110 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:52.640881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:52.640836 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 02 18:34:52.641117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:52.640893 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 02 18:34:52.690342 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:52.690300 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:34:52.690573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:52.690360 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:34:52.695777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:52.695722 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 02 18:34:52.695950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:52.695805 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 02 18:34:54.401835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.401795 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:54.402646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.402622 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:54.403509 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.403485 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:34:54.457725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.457694 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:54.458543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.458516 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:54.459989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.459967 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:34:54.513370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.513338 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:54.513975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.513945 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:34:54.514829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:34:54.514812 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:35:05.948462 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:35:05.948420 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:35:35.093815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:35.093781 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d"] Nov 02 18:35:35.094398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:35.093985 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" containerID="cri-o://73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0" gracePeriod=30 Nov 02 18:35:35.948545 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:35:35.948497 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:35:40.369572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.369504 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:35:40.446462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446423 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.446659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446485 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.446659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446514 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.446659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446560 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.446893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446717 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.446893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446838 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:40.447013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446905 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.447013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446947 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.447013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446962 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:40.447013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.446974 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:35:40.447246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.447026 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.447246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.447059 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo\") pod \"05a60368-c286-4b63-b463-daca81f664eb\" (UID: \"05a60368-c286-4b63-b463-daca81f664eb\") " Nov 02 18:35:40.447246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.447192 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:40.447429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.447260 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:40.448027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.448000 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.448027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.448031 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.448251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.448047 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.448251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.448061 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.448251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.448075 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a60368-c286-4b63-b463-daca81f664eb-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.449116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.449088 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:35:40.449239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.449142 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:40.449309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.449290 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "kube-api-access-c57pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:35:40.449371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.449328 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo" pod "05a60368-c286-4b63-b463-daca81f664eb" (UID: "05a60368-c286-4b63-b463-daca81f664eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:35:40.549605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.549560 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.549605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.549592 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a60368-c286-4b63-b463-daca81f664eb-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.549605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.549605 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a60368-c286-4b63-b463-daca81f664eb-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.549991 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.549619 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57pq\" (UniqueName: \"kubernetes.io/projected/05a60368-c286-4b63-b463-daca81f664eb-kube-api-access-c57pq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:35:40.640603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.640512 5647 generic.go:356] "Generic (PLEG): container finished" podID="05a60368-c286-4b63-b463-daca81f664eb" containerID="73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0" exitCode=0 Nov 02 18:35:40.640603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.640568 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" Nov 02 18:35:40.640603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.640590 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" event={"ID":"05a60368-c286-4b63-b463-daca81f664eb","Type":"ContainerDied","Data":"73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0"} Nov 02 18:35:40.640900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.640671 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d" event={"ID":"05a60368-c286-4b63-b463-daca81f664eb","Type":"ContainerDied","Data":"42e509ea58a0be2e7eed4539016a52d9fd0b9d3aa3e29e80cb37e13804c39dc6"} Nov 02 18:35:40.640900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.640695 5647 scope.go:122] "RemoveContainer" containerID="73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0" Nov 02 18:35:40.652195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.652173 5647 scope.go:122] "RemoveContainer" containerID="4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1" Nov 02 18:35:40.663039 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.663015 5647 scope.go:122] "RemoveContainer" containerID="73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0" Nov 02 18:35:40.663358 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:35:40.663336 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0\": container with ID starting with 73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0 not found: ID does not exist" containerID="73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0" Nov 02 18:35:40.663405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.663373 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0"} err="failed to get container status \"73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0\": rpc error: code = NotFound desc = could not find container \"73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0\": container with ID starting with 73fe4e0aaa1d92fe580cdd25aea1bd2a9d8d057425419781b791b52b2f68e4b0 not found: ID does not exist" Nov 02 18:35:40.663405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.663396 5647 scope.go:122] "RemoveContainer" containerID="4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1" Nov 02 18:35:40.663682 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:35:40.663664 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1\": container with ID starting with 4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1 not found: ID does not exist" containerID="4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1" Nov 02 18:35:40.663762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.663689 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1"} err="failed to get container status \"4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1\": rpc error: code = NotFound desc = could not find container \"4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1\": container with ID starting with 4491b177b8cf9ef0286891683fcb0de0ea03c995c0f7998e3d06a55e0cb802f1 not found: ID does not exist" Nov 02 18:35:40.807983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.807936 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d"] Nov 02 18:35:40.813950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:40.813920 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c70712-openshift-default-d4f6f69dc-9694d"] Nov 02 18:35:41.082861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:41.082775 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a60368-c286-4b63-b463-daca81f664eb" path="/var/lib/kubelet/pods/05a60368-c286-4b63-b463-daca81f664eb/volumes" Nov 02 18:35:52.119964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:35:52.119924 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:36:05.947689 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:36:05.947649 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:36:17.534043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.534003 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz"] Nov 02 18:36:17.534654 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.534290 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" podUID="f151ed30-8d9c-40e2-8425-bb4b4c17b624" containerName="maas-api" containerID="cri-o://048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b" gracePeriod=30 Nov 02 18:36:17.879894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.879869 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:36:17.983163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.983123 5647 generic.go:356] "Generic (PLEG): container finished" podID="f151ed30-8d9c-40e2-8425-bb4b4c17b624" containerID="048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b" exitCode=0 Nov 02 18:36:17.983469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.983441 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" Nov 02 18:36:17.983708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.983683 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" event={"ID":"f151ed30-8d9c-40e2-8425-bb4b4c17b624","Type":"ContainerDied","Data":"048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b"} Nov 02 18:36:17.983892 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.983874 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz" event={"ID":"f151ed30-8d9c-40e2-8425-bb4b4c17b624","Type":"ContainerDied","Data":"6d9773d08ba7c7781dcef55f36707fa5787a3f7b70e7424a4694c7d462ba891f"} Nov 02 18:36:17.984034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.984020 5647 scope.go:122] "RemoveContainer" containerID="048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b" Nov 02 18:36:17.984171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.984147 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs\") pod \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " Nov 02 18:36:17.984243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.984197 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr\" (UniqueName: \"kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr\") pod \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " Nov 02 18:36:17.984316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.984300 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\") pod \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " Nov 02 18:36:17.984431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.984415 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca\") pod \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\" (UID: \"f151ed30-8d9c-40e2-8425-bb4b4c17b624\") " Nov 02 18:36:17.987172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.985341 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca" pod "f151ed30-8d9c-40e2-8425-bb4b4c17b624" (UID: "f151ed30-8d9c-40e2-8425-bb4b4c17b624"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:17.991960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.991923 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls" pod "f151ed30-8d9c-40e2-8425-bb4b4c17b624" (UID: "f151ed30-8d9c-40e2-8425-bb4b4c17b624"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:17.994674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.994628 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs" pod "f151ed30-8d9c-40e2-8425-bb4b4c17b624" (UID: "f151ed30-8d9c-40e2-8425-bb4b4c17b624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:17.994674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:17.994657 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr" pod "f151ed30-8d9c-40e2-8425-bb4b4c17b624" (UID: "f151ed30-8d9c-40e2-8425-bb4b4c17b624"). InnerVolumeSpecName "kube-api-access-z5rlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:18.019900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.019874 5647 scope.go:122] "RemoveContainer" containerID="048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b" Nov 02 18:36:18.020339 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:36:18.020314 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b\": container with ID starting with 048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b not found: ID does not exist" containerID="048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b" Nov 02 18:36:18.020506 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.020475 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b"} err="failed to get container status \"048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b\": rpc error: code = NotFound desc = could not find container \"048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b\": container with ID starting with 048bb62f6bd2edc19860b778f84fa4120afe2c3a634e691b02355711e530e65b not found: ID does not exist" Nov 02 18:36:18.086117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.086081 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:18.086117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.086116 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rlr\" (UniqueName: \"kubernetes.io/projected/f151ed30-8d9c-40e2-8425-bb4b4c17b624-kube-api-access-z5rlr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:18.086377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.086131 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f151ed30-8d9c-40e2-8425-bb4b4c17b624-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:18.086377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.086144 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f151ed30-8d9c-40e2-8425-bb4b4c17b624-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:18.406034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.406000 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz"] Nov 02 18:36:18.408351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:18.408329 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-662a80-6bdc5446d8-42ffz"] Nov 02 18:36:19.082702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:19.082669 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f151ed30-8d9c-40e2-8425-bb4b4c17b624" path="/var/lib/kubelet/pods/f151ed30-8d9c-40e2-8425-bb4b4c17b624/volumes" Nov 02 18:36:35.947710 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:36:35.947668 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:36:40.038510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:40.038478 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp"] Nov 02 18:36:40.039418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:40.039382 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" containerID="cri-o://69296e1f79dbec778936523e0e34c18a1596296bf832db7a3778e339f0b422db" gracePeriod=30 Nov 02 18:36:42.110431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:42.110393 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7"] Nov 02 18:36:42.110952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:42.110627 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" containerID="cri-o://67b49d009f2af1329f0a3a2a883af240d2994160d049af0d908da91289c69195" gracePeriod=30 Nov 02 18:36:45.070027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.069996 5647 patch_prober.go:29] interesting pod/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:36:45.070445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.070056 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:36:45.320044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.319955 5647 generic.go:356] "Generic (PLEG): container finished" podID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerID="69296e1f79dbec778936523e0e34c18a1596296bf832db7a3778e339f0b422db" exitCode=0 Nov 02 18:36:45.320213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.320042 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" event={"ID":"6afb4a63-964c-4903-8372-7f6cdd4022ff","Type":"ContainerDied","Data":"69296e1f79dbec778936523e0e34c18a1596296bf832db7a3778e339f0b422db"} Nov 02 18:36:45.320213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.320112 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" event={"ID":"6afb4a63-964c-4903-8372-7f6cdd4022ff","Type":"ContainerDied","Data":"7015c6ee631d44adbb9e4f42aaf9275434995b087ff582b9108d919e81449b86"} Nov 02 18:36:45.320213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.320132 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7015c6ee631d44adbb9e4f42aaf9275434995b087ff582b9108d919e81449b86" Nov 02 18:36:45.320213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.320147 5647 scope.go:122] "RemoveContainer" containerID="95f30523fabfcd5e091156060ed9a012b62b6dd75db0f89539b7f94832079f72" Nov 02 18:36:45.334171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.334145 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:36:45.417403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417370 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.417607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417410 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.417607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417508 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.417607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417552 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.417848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417700 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.417848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417765 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.417848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417774 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:45.417848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417810 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.418090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417907 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:45.418090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.417946 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.418090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418008 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:45.418090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418031 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:45.418090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418040 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo\") pod \"6afb4a63-964c-4903-8372-7f6cdd4022ff\" (UID: \"6afb4a63-964c-4903-8372-7f6cdd4022ff\") " Nov 02 18:36:45.418377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418195 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:45.418757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418719 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.418862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418763 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6afb4a63-964c-4903-8372-7f6cdd4022ff-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.418862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418779 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.418862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418793 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.418862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.418810 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.420952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.420912 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:45.421051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.420981 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:36:45.421051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.421013 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:45.421051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.421044 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd" pod "6afb4a63-964c-4903-8372-7f6cdd4022ff" (UID: "6afb4a63-964c-4903-8372-7f6cdd4022ff"). InnerVolumeSpecName "kube-api-access-jm5bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:45.520344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.520306 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.520344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.520339 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm5bd\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-kube-api-access-jm5bd\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.520561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.520354 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:45.520561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:45.520369 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6afb4a63-964c-4903-8372-7f6cdd4022ff-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:46.328214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.328123 5647 generic.go:356] "Generic (PLEG): container finished" podID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerID="7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226" exitCode=0 Nov 02 18:36:46.328214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.328194 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" event={"ID":"b75529ed-454c-417f-8bc5-4b6bf3840a72","Type":"ContainerDied","Data":"7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226"} Nov 02 18:36:46.329322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.328777 5647 scope.go:122] "RemoveContainer" containerID="7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226" Nov 02 18:36:46.330011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.329980 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp" Nov 02 18:36:46.330686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.330648 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:36:46.330794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.330757 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:36:46.330846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.330797 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:36:46.678933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.678896 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp"] Nov 02 18:36:46.683526 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:46.683496 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6cf852-openshift-default-7d57b456df-mmqjp"] Nov 02 18:36:47.081432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.081349 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" path="/var/lib/kubelet/pods/6afb4a63-964c-4903-8372-7f6cdd4022ff/volumes" Nov 02 18:36:47.344485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.344068 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" event={"ID":"b75529ed-454c-417f-8bc5-4b6bf3840a72","Type":"ContainerStarted","Data":"38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d"} Nov 02 18:36:47.347192 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.347159 5647 generic.go:356] "Generic (PLEG): container finished" podID="aafb611c-d65f-447d-b126-881d5e3749f4" containerID="67b49d009f2af1329f0a3a2a883af240d2994160d049af0d908da91289c69195" exitCode=0 Nov 02 18:36:47.347356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.347222 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" event={"ID":"aafb611c-d65f-447d-b126-881d5e3749f4","Type":"ContainerDied","Data":"67b49d009f2af1329f0a3a2a883af240d2994160d049af0d908da91289c69195"} Nov 02 18:36:47.347482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.347310 5647 scope.go:122] "RemoveContainer" containerID="f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e" Nov 02 18:36:47.390892 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:36:47.390868 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e\": container with ID starting with f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e not found: ID does not exist" containerID="f77b04e35baea7b5a5f6c10709c87bba8653542c3dbdc967c0005edbb6b5483e" Nov 02 18:36:47.394319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.394301 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:36:47.544431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544391 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544453 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544518 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544601 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544686 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544714 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544713 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544723 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544770 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544804 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.544868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.544851 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket\") pod \"aafb611c-d65f-447d-b126-881d5e3749f4\" (UID: \"aafb611c-d65f-447d-b126-881d5e3749f4\") " Nov 02 18:36:47.545343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545312 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:47.545542 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545496 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:47.545805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545564 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:47.545919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545895 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.545919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545919 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.546127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545930 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.546127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545939 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aafb611c-d65f-447d-b126-881d5e3749f4-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.546127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.545948 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.548061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.547898 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:36:47.548061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.547977 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:36:47.548061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.548028 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:47.549548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.549530 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw" pod "aafb611c-d65f-447d-b126-881d5e3749f4" (UID: "aafb611c-d65f-447d-b126-881d5e3749f4"). InnerVolumeSpecName "kube-api-access-zfcvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:47.587524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.587490 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:36:47.587775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.587754 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 02 18:36:47.587851 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.587800 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 02 18:36:47.647370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.647267 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcvw\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-kube-api-access-zfcvw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.647370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.647302 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aafb611c-d65f-447d-b126-881d5e3749f4-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.647370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.647317 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aafb611c-d65f-447d-b126-881d5e3749f4-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:47.647370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:47.647329 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aafb611c-d65f-447d-b126-881d5e3749f4-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:48.354550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.354516 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" Nov 02 18:36:48.355133 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.354562 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7" event={"ID":"aafb611c-d65f-447d-b126-881d5e3749f4","Type":"ContainerDied","Data":"766c0bd64172192723b30221e4e235f1358fbed9bd96bcdda4c9d17200f99b86"} Nov 02 18:36:48.355133 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.354637 5647 scope.go:122] "RemoveContainer" containerID="67b49d009f2af1329f0a3a2a883af240d2994160d049af0d908da91289c69195" Nov 02 18:36:48.531490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.531455 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7"] Nov 02 18:36:48.581376 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.581335 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-546881-openshift-default-74c4855bbc-tqgb7"] Nov 02 18:36:48.587055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.587030 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 02 18:36:48.587195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:48.587089 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 02 18:36:49.081694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:49.081657 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" path="/var/lib/kubelet/pods/aafb611c-d65f-447d-b126-881d5e3749f4/volumes" Nov 02 18:36:49.587053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:49.587020 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 02 18:36:49.587571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:49.587076 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 02 18:36:50.587170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:50.587138 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 02 18:36:50.587609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:50.587188 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 02 18:36:51.366070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.366027 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk"] Nov 02 18:36:51.366695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.366663 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" podUID="400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" containerName="maas-api" containerID="cri-o://b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0" gracePeriod=30 Nov 02 18:36:51.587989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.587953 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 02 18:36:51.588366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.588018 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 02 18:36:51.901370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.900963 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:36:51.920288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.918918 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs\") pod \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " Nov 02 18:36:51.920288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.918977 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\") pod \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " Nov 02 18:36:51.920288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.919022 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca\") pod \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " Nov 02 18:36:51.920288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.919211 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k\" (UniqueName: \"kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k\") pod \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\" (UID: \"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d\") " Nov 02 18:36:51.921403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.921352 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca" pod "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" (UID: "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:36:51.964136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.963141 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs" pod "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" (UID: "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:51.965392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.965344 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls" pod "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" (UID: "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:36:51.966457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:51.966428 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k" pod "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" (UID: "400d7ef0-ba41-4e7f-9ea9-ed886038ec0d"). InnerVolumeSpecName "kube-api-access-pqp5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:36:52.030759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.025970 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqp5k\" (UniqueName: \"kubernetes.io/projected/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-kube-api-access-pqp5k\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:52.030759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.026008 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:52.030759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.026025 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:52.030759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.026038 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:36:52.394161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.394090 5647 generic.go:356] "Generic (PLEG): container finished" podID="400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" containerID="b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0" exitCode=0 Nov 02 18:36:52.394161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.394145 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" Nov 02 18:36:52.394574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.394181 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" event={"ID":"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d","Type":"ContainerDied","Data":"b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0"} Nov 02 18:36:52.394574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.394240 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk" event={"ID":"400d7ef0-ba41-4e7f-9ea9-ed886038ec0d","Type":"ContainerDied","Data":"8157933fbf17539fb91229fb574df17c686df08472836bd925ccad32ad1a9a9d"} Nov 02 18:36:52.394574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.394272 5647 scope.go:122] "RemoveContainer" containerID="b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0" Nov 02 18:36:52.433167 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.431994 5647 scope.go:122] "RemoveContainer" containerID="b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0" Nov 02 18:36:52.437757 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:36:52.435118 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0\": container with ID starting with b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0 not found: ID does not exist" containerID="b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0" Nov 02 18:36:52.437757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.435161 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0"} err="failed to get container status \"b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0\": rpc error: code = NotFound desc = could not find container \"b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0\": container with ID starting with b66af33af4cd4b313309f41382d0e2b990cd7fa89d52286520cac7576df697b0 not found: ID does not exist" Nov 02 18:36:52.593174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.593131 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 02 18:36:52.593716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.593209 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 02 18:36:52.599802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.599725 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk"] Nov 02 18:36:52.616780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.604241 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a51e27-5c669f84bf-q59kk"] Nov 02 18:36:52.848356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:52.848318 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:36:53.081602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:53.081564 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" path="/var/lib/kubelet/pods/400d7ef0-ba41-4e7f-9ea9-ed886038ec0d/volumes" Nov 02 18:36:53.592072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:53.592038 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:36:53.593217 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:53.593196 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:36:54.449049 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:36:54.449015 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:37:05.948034 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:37:05.947995 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:37:09.689086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689042 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn"] Nov 02 18:37:09.689670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689650 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f151ed30-8d9c-40e2-8425-bb4b4c17b624" containerName="maas-api" Nov 02 18:37:09.689670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689669 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f151ed30-8d9c-40e2-8425-bb4b4c17b624" containerName="maas-api" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689681 5647 cpu_manager.go:414] "containerMap: removing container" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689687 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689719 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689724 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689755 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689762 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689770 5647 cpu_manager.go:414] "containerMap: removing container" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689775 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689785 5647 cpu_manager.go:414] "containerMap: removing container" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689790 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689798 5647 cpu_manager.go:414] "containerMap: removing container" podUID="400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" containerName="maas-api" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689803 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" containerName="maas-api" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689816 5647 cpu_manager.go:414] "containerMap: removing container" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" Nov 02 18:37:09.689870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689823 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689923 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689932 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="400d7ef0-ba41-4e7f-9ea9-ed886038ec0d" containerName="maas-api" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689939 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689948 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f151ed30-8d9c-40e2-8425-bb4b4c17b624" containerName="maas-api" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689954 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafb611c-d65f-447d-b126-881d5e3749f4" containerName="istio-proxy" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689966 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" Nov 02 18:37:09.690343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.689972 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6afb4a63-964c-4903-8372-7f6cdd4022ff" containerName="istio-proxy" Nov 02 18:37:09.694931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.694910 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.759074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.759039 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4fe24d-openshift-default-dockercfg-tbxqp\"" Nov 02 18:37:09.842638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.842583 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.843674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.843646 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.843881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.843855 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.846967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.846944 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.847240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.847208 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.847339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.847292 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.847339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.847329 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.847455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.847358 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4hm\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.847455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.847432 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.854921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.854872 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn"] Nov 02 18:37:09.972785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.972661 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.972785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.972763 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.972799 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.972837 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.973259 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.973331 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973458 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.973408 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973458 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.973447 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.973480 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4hm\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.973585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.973550 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.974175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.974145 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.974306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.974281 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.974445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.974419 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.974696 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.974669 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.977827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.977800 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:09.978224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:09.978206 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:10.032279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.032243 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:10.032474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.032449 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4hm\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm\") pod \"e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:10.309782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.309673 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:10.737862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.737835 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn"] Nov 02 18:37:10.741488 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:37:10.741461 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecb6c0b_afc3_46a0_8f03_3acd652fbfe0.slice/crio-33117dd955497b3b5c67277b0fef7748036fc2e99df4bc3d77c3d166b3e7ba9c WatchSource:0}: Error finding container 33117dd955497b3b5c67277b0fef7748036fc2e99df4bc3d77c3d166b3e7ba9c: Status 404 returned error can't find the container with id 33117dd955497b3b5c67277b0fef7748036fc2e99df4bc3d77c3d166b3e7ba9c Nov 02 18:37:10.744934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.744899 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:37:10.745033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.744971 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:37:10.745033 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:10.745009 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:37:11.517352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:11.517260 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" event={"ID":"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0","Type":"ContainerStarted","Data":"fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b"} Nov 02 18:37:11.517352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:11.517348 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" event={"ID":"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0","Type":"ContainerStarted","Data":"33117dd955497b3b5c67277b0fef7748036fc2e99df4bc3d77c3d166b3e7ba9c"} Nov 02 18:37:11.739665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:11.739596 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podStartSLOduration=2.73958044 podStartE2EDuration="2.73958044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:37:11.686698125 +0000 UTC m=+4418.760225327" watchObservedRunningTime="2026-11-02 18:37:11.73958044 +0000 UTC m=+4418.813107713" Nov 02 18:37:12.310411 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:12.310371 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:12.316846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:12.316810 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:12.525065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:12.525023 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:12.526145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:12.526127 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:37:35.948185 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:37:35.948143 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:37:43.917304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:37:43.917272 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:38:05.960439 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:38:05.960396 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:38:14.865123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:14.865085 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2"] Nov 02 18:38:14.865679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:14.865312 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" containerID="cri-o://ab9b0db6178e7b7b78ca3e6af1cef3cef66ebfdb0b6cd3fa142026b3fb26fa51" gracePeriod=30 Nov 02 18:38:15.257152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:15.257076 5647 patch_prober.go:29] interesting pod/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:38:15.257152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:15.257129 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:38:20.073283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.073249 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" event={"ID":"f393f960-71c5-4ca2-bd2b-d7d1d49b9374","Type":"ContainerDied","Data":"ab9b0db6178e7b7b78ca3e6af1cef3cef66ebfdb0b6cd3fa142026b3fb26fa51"} Nov 02 18:38:20.073887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.073304 5647 scope.go:122] "RemoveContainer" containerID="5e5291c127e6cdfc5222440d52ab87e9c5c02cb256dd24cb47d08f8c49688d65" Nov 02 18:38:20.073887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.073245 5647 generic.go:356] "Generic (PLEG): container finished" podID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerID="ab9b0db6178e7b7b78ca3e6af1cef3cef66ebfdb0b6cd3fa142026b3fb26fa51" exitCode=0 Nov 02 18:38:20.138515 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.138454 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:38:20.237084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237052 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237125 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237183 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237228 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237283 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237319 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237368 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237370 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:38:20.237407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237391 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237414 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt\") pod \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\" (UID: \"f393f960-71c5-4ca2-bd2b-d7d1d49b9374\") " Nov 02 18:38:20.237673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237469 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:38:20.237821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237687 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:38:20.237910 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237892 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:38:20.237979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.237918 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:38:20.239018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.238988 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.239163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.239022 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.239163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.239041 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.239163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.239057 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.239163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.239074 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.240630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.240605 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:38:20.241013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.240991 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:38:20.241309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.241291 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:38:20.241564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.241542 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt" pod "f393f960-71c5-4ca2-bd2b-d7d1d49b9374" (UID: "f393f960-71c5-4ca2-bd2b-d7d1d49b9374"). InnerVolumeSpecName "kube-api-access-gd8tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:38:20.340281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.340249 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.340281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.340281 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.340508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.340297 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8tt\" (UniqueName: \"kubernetes.io/projected/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-kube-api-access-gd8tt\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:20.340508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:20.340311 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f393f960-71c5-4ca2-bd2b-d7d1d49b9374-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:21.081360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:21.081328 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" Nov 02 18:38:21.082303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:21.082276 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2" event={"ID":"f393f960-71c5-4ca2-bd2b-d7d1d49b9374","Type":"ContainerDied","Data":"c9cb183a67c8ed48b85a78b23f0d73f7c983a5c8c1a5524af2ce5d8f297385c4"} Nov 02 18:38:21.082441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:21.082335 5647 scope.go:122] "RemoveContainer" containerID="ab9b0db6178e7b7b78ca3e6af1cef3cef66ebfdb0b6cd3fa142026b3fb26fa51" Nov 02 18:38:21.299107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:21.299073 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2"] Nov 02 18:38:21.349508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:21.349477 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a51e27-openshift-default-7f579fb6dc-j74q2"] Nov 02 18:38:23.085853 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:23.085804 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" path="/var/lib/kubelet/pods/f393f960-71c5-4ca2-bd2b-d7d1d49b9374/volumes" Nov 02 18:38:34.794377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:34.794344 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:38:34.794928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:34.794591 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:38:35.947866 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:38:35.947794 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:38:37.464147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:37.464110 5647 scope.go:122] "RemoveContainer" containerID="69296e1f79dbec778936523e0e34c18a1596296bf832db7a3778e339f0b422db" Nov 02 18:38:45.698067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:45.697979 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2"] Nov 02 18:38:45.698546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:45.698229 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" podUID="a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" containerName="maas-api" containerID="cri-o://822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0" gracePeriod=30 Nov 02 18:38:46.033723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.033688 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:38:46.170438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.170402 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\") pod \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " Nov 02 18:38:46.171298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.171266 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs\") pod \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " Nov 02 18:38:46.171464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.171329 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9\" (UniqueName: \"kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9\") pod \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " Nov 02 18:38:46.171464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.171381 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca\") pod \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\" (UID: \"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9\") " Nov 02 18:38:46.172415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.172148 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca" pod "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" (UID: "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:38:46.173077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.173043 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls" pod "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" (UID: "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:38:46.174162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.174131 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9" pod "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" (UID: "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9"). InnerVolumeSpecName "kube-api-access-246m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:38:46.174437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.174418 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs" pod "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" (UID: "a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:38:46.309809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.294039 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:46.309809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.294077 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246m9\" (UniqueName: \"kubernetes.io/projected/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-kube-api-access-246m9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:46.309809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.294097 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:46.309809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.294111 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:38:46.348568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.348529 5647 generic.go:356] "Generic (PLEG): container finished" podID="a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" containerID="822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0" exitCode=0 Nov 02 18:38:46.348807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.348622 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" event={"ID":"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9","Type":"ContainerDied","Data":"822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0"} Nov 02 18:38:46.348807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.348675 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" event={"ID":"a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9","Type":"ContainerDied","Data":"a917a39ac5b39887b1166640709aaf95365ecb017723a3600285fe48c6b06a2e"} Nov 02 18:38:46.348807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.348706 5647 scope.go:122] "RemoveContainer" containerID="822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0" Nov 02 18:38:46.349056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.348881 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2" Nov 02 18:38:46.365984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.365962 5647 scope.go:122] "RemoveContainer" containerID="822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0" Nov 02 18:38:46.366281 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:38:46.366259 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0\": container with ID starting with 822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0 not found: ID does not exist" containerID="822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0" Nov 02 18:38:46.366357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.366293 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0"} err="failed to get container status \"822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0\": rpc error: code = NotFound desc = could not find container \"822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0\": container with ID starting with 822c445b157118f3f32d1f91ea59a9a7a29feab051b04a9c772b36e2a0925fd0 not found: ID does not exist" Nov 02 18:38:46.516682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.516641 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2"] Nov 02 18:38:46.522207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:46.522168 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-608907-55b54f9d7d-d5nr2"] Nov 02 18:38:47.082038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:38:47.081999 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" path="/var/lib/kubelet/pods/a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9/volumes" Nov 02 18:39:03.549298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:03.549259 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:39:06.113998 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:39:06.113958 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:39:07.294760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:07.294045 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 18:39:13.566908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:13.566873 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:39:13.567446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:13.566960 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:39:15.669899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:15.669866 5647 generic.go:356] "Generic (PLEG): container finished" podID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerID="fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b" exitCode=0 Nov 02 18:39:15.670379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:15.669944 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" event={"ID":"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0","Type":"ContainerDied","Data":"fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b"} Nov 02 18:39:15.670379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:15.670363 5647 scope.go:122] "RemoveContainer" containerID="fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b" Nov 02 18:39:15.672017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:15.671983 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:39:15.672156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:15.672071 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:39:15.672156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:15.672118 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:39:16.678545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:16.678501 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" event={"ID":"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0","Type":"ContainerStarted","Data":"0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf"} Nov 02 18:39:17.327951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:17.327899 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:39:17.328162 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:17.328139 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 02 18:39:17.328228 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:17.328203 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 02 18:39:18.318164 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:18.318129 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 02 18:39:18.318815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:18.318777 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 02 18:39:19.326914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:19.326873 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 02 18:39:19.327456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:19.326932 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 02 18:39:20.321206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:20.321163 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 02 18:39:20.321532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:20.321502 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 02 18:39:21.311139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:21.311101 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" start-of-body= Nov 02 18:39:21.311784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:21.311724 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.30:15021/healthz/ready\": dial tcp 10.134.1.30:15021: connect: connection refused" Nov 02 18:39:22.314460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:22.314426 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:39:22.314994 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:22.314856 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:39:22.315629 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:39:22.315610 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:39:35.948013 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:39:35.947973 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:40:00.908581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.908539 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp"] Nov 02 18:40:00.909189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909171 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" containerName="maas-api" Nov 02 18:40:00.909240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909191 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" containerName="maas-api" Nov 02 18:40:00.909240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909212 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" Nov 02 18:40:00.909240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909218 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" Nov 02 18:40:00.909349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909241 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" Nov 02 18:40:00.909349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909246 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" Nov 02 18:40:00.909417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909353 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76c6de9-c11f-43a6-9d9f-baa3cb01d7d9" containerName="maas-api" Nov 02 18:40:00.909417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909366 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a60368-c286-4b63-b463-daca81f664eb" containerName="istio-proxy" Nov 02 18:40:00.909417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909374 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" Nov 02 18:40:00.909417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.909384 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f393f960-71c5-4ca2-bd2b-d7d1d49b9374" containerName="istio-proxy" Nov 02 18:40:00.913091 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.913075 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:00.968848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:00.968818 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp"] Nov 02 18:40:01.016397 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.016357 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:40:01.016609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.016411 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:40:01.019132 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.019100 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.019266 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.019159 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.019310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.019269 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29tl7\" (UniqueName: \"kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.120609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.120571 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.120868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.120620 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.120868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.120654 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29tl7\" (UniqueName: \"kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.121299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.121272 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.123267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.123247 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.178618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.178542 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29tl7\" (UniqueName: \"kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7\") pod \"maas-api-revoke-sub-5c1d6abf-wqhxp\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.227073 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.227035 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:01.604339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:01.604314 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp"] Nov 02 18:40:01.606914 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:40:01.606881 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b20b57_e3f1_4069_ae2d_45b2a5957f54.slice/crio-ad4d077447c47112775b58e9902932e480543858ea3cbeccd3f18f0399c6e0f7 WatchSource:0}: Error finding container ad4d077447c47112775b58e9902932e480543858ea3cbeccd3f18f0399c6e0f7: Status 404 returned error can't find the container with id ad4d077447c47112775b58e9902932e480543858ea3cbeccd3f18f0399c6e0f7 Nov 02 18:40:02.001283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:02.001187 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" event={"ID":"30b20b57-e3f1-4069-ae2d-45b2a5957f54","Type":"ContainerStarted","Data":"ad4d077447c47112775b58e9902932e480543858ea3cbeccd3f18f0399c6e0f7"} Nov 02 18:40:03.009371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:03.009334 5647 generic.go:356] "Generic (PLEG): container finished" podID="30b20b57-e3f1-4069-ae2d-45b2a5957f54" containerID="1ed266a3d101f96036f12df81bce5ab7bf165b6ec4ec7275be2da820532d1621" exitCode=0 Nov 02 18:40:03.009819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:03.009385 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" event={"ID":"30b20b57-e3f1-4069-ae2d-45b2a5957f54","Type":"ContainerDied","Data":"1ed266a3d101f96036f12df81bce5ab7bf165b6ec4ec7275be2da820532d1621"} Nov 02 18:40:04.176671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.176641 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:04.280896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.270303 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca\") pod \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " Nov 02 18:40:04.280896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.270496 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca\") pod \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " Nov 02 18:40:04.280896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.270557 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7\" (UniqueName: \"kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7\") pod \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\" (UID: \"30b20b57-e3f1-4069-ae2d-45b2a5957f54\") " Nov 02 18:40:04.315786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.313643 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca" pod "30b20b57-e3f1-4069-ae2d-45b2a5957f54" (UID: "30b20b57-e3f1-4069-ae2d-45b2a5957f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:40:04.315786 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.313955 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7" pod "30b20b57-e3f1-4069-ae2d-45b2a5957f54" (UID: "30b20b57-e3f1-4069-ae2d-45b2a5957f54"). InnerVolumeSpecName "kube-api-access-29tl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:40:04.322817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.320250 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca" pod "30b20b57-e3f1-4069-ae2d-45b2a5957f54" (UID: "30b20b57-e3f1-4069-ae2d-45b2a5957f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:40:04.372860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.372828 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:40:04.372860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.372858 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29tl7\" (UniqueName: \"kubernetes.io/projected/30b20b57-e3f1-4069-ae2d-45b2a5957f54-kube-api-access-29tl7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:40:04.373066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:04.372871 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30b20b57-e3f1-4069-ae2d-45b2a5957f54-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:40:05.023517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:05.023478 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" event={"ID":"30b20b57-e3f1-4069-ae2d-45b2a5957f54","Type":"ContainerDied","Data":"ad4d077447c47112775b58e9902932e480543858ea3cbeccd3f18f0399c6e0f7"} Nov 02 18:40:05.023517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:05.023499 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp" Nov 02 18:40:05.023775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:05.023530 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad4d077447c47112775b58e9902932e480543858ea3cbeccd3f18f0399c6e0f7" Nov 02 18:40:05.947522 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:40:05.947485 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:40:15.362849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:40:15.362756 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:40:35.995892 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:40:35.995846 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:41:05.947743 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:41:05.947691 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:41:19.294751 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:19.294702 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:41:35.949325 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:41:35.949277 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:41:42.850819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.850704 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh"] Nov 02 18:41:42.851518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.851496 5647 cpu_manager.go:414] "containerMap: removing container" podUID="30b20b57-e3f1-4069-ae2d-45b2a5957f54" containerName="revoke-keys" Nov 02 18:41:42.851625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.851520 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b20b57-e3f1-4069-ae2d-45b2a5957f54" containerName="revoke-keys" Nov 02 18:41:42.851767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.851725 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b20b57-e3f1-4069-ae2d-45b2a5957f54" containerName="revoke-keys" Nov 02 18:41:42.855276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.855255 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:42.942767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.916888 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8f40154e-openshift-default-dockercfg-2jxtq\"" Nov 02 18:41:42.955367 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:42.943984 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124250 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124328 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124371 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124455 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124499 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mv5q\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124547 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124576 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.124663 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.124887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.121284 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh"] Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229217 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229287 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229370 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229459 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229533 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229590 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229631 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.230755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.229718 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.235543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.235507 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mv5q\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.238051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.237364 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.238051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.238002 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.238298 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.238240 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.238478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.238360 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.238604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.238575 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.240810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.240715 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.241689 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.241653 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.294309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.294277 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.295534 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.295500 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mv5q\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q\") pod \"e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.469288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.469203 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:43.899544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.899515 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh"] Nov 02 18:41:43.900483 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:41:43.900457 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f904cb_7a91_46e9_9b1b_64be91044bd9.slice/crio-4321296ea7bf86716d01446acc33165fc556361519cb97173b254a4d615b4d33 WatchSource:0}: Error finding container 4321296ea7bf86716d01446acc33165fc556361519cb97173b254a4d615b4d33: Status 404 returned error can't find the container with id 4321296ea7bf86716d01446acc33165fc556361519cb97173b254a4d615b4d33 Nov 02 18:41:43.903251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.903219 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:41:43.903337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.903290 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:41:43.903337 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:43.903320 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:41:44.788423 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:44.788385 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" event={"ID":"61f904cb-7a91-46e9-9b1b-64be91044bd9","Type":"ContainerStarted","Data":"f14dd11c4a166a2f3bd9bc5e5c12d09950202c5c39fc538b439337de343fedca"} Nov 02 18:41:44.788620 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:44.788475 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" event={"ID":"61f904cb-7a91-46e9-9b1b-64be91044bd9","Type":"ContainerStarted","Data":"4321296ea7bf86716d01446acc33165fc556361519cb97173b254a4d615b4d33"} Nov 02 18:41:45.008216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:45.008158 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podStartSLOduration=3.008143494 podStartE2EDuration="3.008143494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:41:44.955862984 +0000 UTC m=+4692.029390188" watchObservedRunningTime="2026-11-02 18:41:45.008143494 +0000 UTC m=+4692.081670702" Nov 02 18:41:45.469835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:45.469798 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:45.476314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:45.476283 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:45.794576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:45.794485 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:45.795660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:45.795640 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:41:58.568490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.568449 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn"] Nov 02 18:41:58.573178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.573156 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.596648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.596608 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt6x\" (UniqueName: \"kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x\") pod \"payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.596844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.596747 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.631919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.631884 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn"] Nov 02 18:41:58.704880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.704846 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8f40154e\"" Nov 02 18:41:58.705225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.705206 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8f40154e-dockercfg-mnglh\"" Nov 02 18:41:58.732806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.721751 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt6x\" (UniqueName: \"kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x\") pod \"payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.733235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.733206 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.747184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.747131 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx"] Nov 02 18:41:58.748970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.748923 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.802395 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.802259 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:58.809860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.809817 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx"] Nov 02 18:41:58.820541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.820458 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt6x\" (UniqueName: \"kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x\") pod \"payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.888272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.888243 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:41:58.951078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.950677 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftlh\" (UniqueName: \"kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh\") pod \"payload-processing-e2e-mt-8f40154e-56f7694565-fndwx\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:58.951078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:58.950943 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f40154e-56f7694565-fndwx\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.052443 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.052404 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f40154e-56f7694565-fndwx\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.052629 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.052525 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftlh\" (UniqueName: \"kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh\") pod \"payload-processing-e2e-mt-8f40154e-56f7694565-fndwx\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.053131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.053112 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume\") pod \"payload-processing-e2e-mt-8f40154e-56f7694565-fndwx\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.110868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.110831 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftlh\" (UniqueName: \"kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh\") pod \"payload-processing-e2e-mt-8f40154e-56f7694565-fndwx\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.140983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.140947 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.267550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.267521 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn"] Nov 02 18:41:59.268956 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:41:59.268928 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d6e7cc_d508_4f3b_91ed_c7cafdcba7b6.slice/crio-f0549694ac60f7fc0b735ab9de2547e46883a4c227fdd55f053c615556281ee6 WatchSource:0}: Error finding container f0549694ac60f7fc0b735ab9de2547e46883a4c227fdd55f053c615556281ee6: Status 404 returned error can't find the container with id f0549694ac60f7fc0b735ab9de2547e46883a4c227fdd55f053c615556281ee6 Nov 02 18:41:59.519778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.519749 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx"] Nov 02 18:41:59.521171 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:41:59.521137 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbc72d7_bb45_4a9a_a69b_39c7e8503aaf.slice/crio-b854e3eb8ce46ad41e2a10d527c58f9f735d1d1f00e65237e6f62bb1919b13bb WatchSource:0}: Error finding container b854e3eb8ce46ad41e2a10d527c58f9f735d1d1f00e65237e6f62bb1919b13bb: Status 404 returned error can't find the container with id b854e3eb8ce46ad41e2a10d527c58f9f735d1d1f00e65237e6f62bb1919b13bb Nov 02 18:41:59.891166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.891128 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" event={"ID":"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf","Type":"ContainerStarted","Data":"8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd"} Nov 02 18:41:59.891624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.891230 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" event={"ID":"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf","Type":"ContainerStarted","Data":"b854e3eb8ce46ad41e2a10d527c58f9f735d1d1f00e65237e6f62bb1919b13bb"} Nov 02 18:41:59.891624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.891478 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:41:59.892921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.892894 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" event={"ID":"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6","Type":"ContainerStarted","Data":"9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e"} Nov 02 18:41:59.893053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.892945 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" event={"ID":"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6","Type":"ContainerStarted","Data":"f0549694ac60f7fc0b735ab9de2547e46883a4c227fdd55f053c615556281ee6"} Nov 02 18:41:59.893053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:41:59.892985 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:42:00.009356 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:42:00.009283 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" podStartSLOduration=2.009268102 podStartE2EDuration="2.009268102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:42:00.006838886 +0000 UTC m=+4707.080366091" watchObservedRunningTime="2026-11-02 18:42:00.009268102 +0000 UTC m=+4707.082795352" Nov 02 18:42:05.948493 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:42:05.948451 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:42:10.901782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:42:10.901724 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:42:10.902216 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:42:10.901874 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:42:11.019174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:42:11.019116 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" podStartSLOduration=13.019100705 podStartE2EDuration="13.019100705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:42:00.152063183 +0000 UTC m=+4707.225590389" watchObservedRunningTime="2026-11-02 18:42:11.019100705 +0000 UTC m=+4718.092627953" Nov 02 18:42:15.602773 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:42:15.602725 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:42:35.947862 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:42:35.947824 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:43:02.848430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.848397 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc"] Nov 02 18:43:02.852817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.852793 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:02.909140 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.909112 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc"] Nov 02 18:43:02.947594 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.947547 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrxbg\" (UniqueName: \"kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:02.947832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.947609 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:02.947832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.947728 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:02.947989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.947833 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:02.956449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:02.956417 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8f40154e-serving-cert\"" Nov 02 18:43:03.049541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.049501 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.049782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.049575 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.049782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.049611 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.049885 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:43:03.049826 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-serving-cert: secret "maas-api-e2e-mt-8f40154e-serving-cert" not found Nov 02 18:43:03.049885 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.049859 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrxbg\" (UniqueName: \"kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.049956 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:43:03.049897 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls podName:1136cf8f-37c0-4b8a-b231-3cb9c699ae39 nodeName:}" failed. No retries permitted until 2026-11-02 18:43:03.549879145 +0000 UTC m=+4770.623406327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls") pod "maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" (UID: "1136cf8f-37c0-4b8a-b231-3cb9c699ae39") : secret "maas-api-e2e-mt-8f40154e-serving-cert" not found Nov 02 18:43:03.050257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.050235 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.052340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.052316 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.131909 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.131783 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrxbg\" (UniqueName: \"kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.559430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.559336 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.562008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.561977 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\") pod \"maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:03.767650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:03.767617 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:04.147720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:04.147686 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc"] Nov 02 18:43:04.149071 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:43:04.149040 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1136cf8f_37c0_4b8a_b231_3cb9c699ae39.slice/crio-9384c71efcab6f2aadf92c5ac8fba63533fa962f0fa523d481da777858964cd1 WatchSource:0}: Error finding container 9384c71efcab6f2aadf92c5ac8fba63533fa962f0fa523d481da777858964cd1: Status 404 returned error can't find the container with id 9384c71efcab6f2aadf92c5ac8fba63533fa962f0fa523d481da777858964cd1 Nov 02 18:43:04.150462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:04.150443 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:43:04.345161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:04.345119 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" event={"ID":"1136cf8f-37c0-4b8a-b231-3cb9c699ae39","Type":"ContainerStarted","Data":"9384c71efcab6f2aadf92c5ac8fba63533fa962f0fa523d481da777858964cd1"} Nov 02 18:43:05.352961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:05.352924 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" event={"ID":"1136cf8f-37c0-4b8a-b231-3cb9c699ae39","Type":"ContainerStarted","Data":"d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001"} Nov 02 18:43:05.353408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:05.353060 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:05.469670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:05.469616 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" podStartSLOduration=2.574601817 podStartE2EDuration="3.469600501s" totalImagesPullingTime="894.998684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:43:05.468693438 +0000 UTC m=+4772.542220640" watchObservedRunningTime="2026-11-02 18:43:05.469600501 +0000 UTC m=+4772.543127703" Nov 02 18:43:05.948497 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:43:05.948461 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:43:07.686107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:07.686071 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:43:11.365895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:11.365823 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:43:34.863445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:34.863414 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:43:34.868996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:34.868978 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:43:35.955991 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:43:35.955957 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:43:40.596294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:40.596255 5647 generic.go:356] "Generic (PLEG): container finished" podID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerID="f14dd11c4a166a2f3bd9bc5e5c12d09950202c5c39fc538b439337de343fedca" exitCode=0 Nov 02 18:43:40.596778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:40.596333 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" event={"ID":"61f904cb-7a91-46e9-9b1b-64be91044bd9","Type":"ContainerDied","Data":"f14dd11c4a166a2f3bd9bc5e5c12d09950202c5c39fc538b439337de343fedca"} Nov 02 18:43:40.596932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:40.596913 5647 scope.go:122] "RemoveContainer" containerID="f14dd11c4a166a2f3bd9bc5e5c12d09950202c5c39fc538b439337de343fedca" Nov 02 18:43:40.598682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:40.598651 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:43:40.598779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:40.598728 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:43:40.598836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:40.598801 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:43:41.607771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:41.607243 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" event={"ID":"61f904cb-7a91-46e9-9b1b-64be91044bd9","Type":"ContainerStarted","Data":"b88b2cbe1ed82a372772084d43b350b3814ffb77e77e1622d2e65e55d80a3151"} Nov 02 18:43:42.469914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:42.469884 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:43:42.471181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:42.471147 5647 patch_prober.go:29] interesting pod/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 02 18:43:42.471318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:42.471202 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 02 18:43:43.470282 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:43.470245 5647 patch_prober.go:29] interesting pod/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 02 18:43:43.470703 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:43.470302 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 02 18:43:44.470541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:44.470505 5647 patch_prober.go:29] interesting pod/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 02 18:43:44.470979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:44.470563 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 02 18:43:45.470230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:45.470186 5647 patch_prober.go:29] interesting pod/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 02 18:43:45.470453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:45.470270 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 02 18:43:45.794796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:45.794679 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:43:46.470149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:46.470103 5647 patch_prober.go:29] interesting pod/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" start-of-body= Nov 02 18:43:46.470353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:46.470167 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.32:15021/healthz/ready\": dial tcp 10.134.1.32:15021: connect: connection refused" Nov 02 18:43:47.475105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:47.475073 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:43:47.476318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:43:47.476296 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:44:05.947554 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:44:05.947516 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:44:06.994878 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:44:06.994843 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:44:35.947253 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:44:35.947215 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:44:51.558492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:44:51.558406 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:45:00.167146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.167110 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw"] Nov 02 18:45:00.171313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.171285 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.174295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.174265 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8"] Nov 02 18:45:00.179411 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.179390 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw"] Nov 02 18:45:00.179668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.179651 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.183749 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.183718 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.218243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.218209 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw"] Nov 02 18:45:00.222444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.222422 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd"] Nov 02 18:45:00.222577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.222532 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.226559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.226538 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.227987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.227965 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:45:00.229585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.229560 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65"] Nov 02 18:45:00.232530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.232510 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.232623 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.232539 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rz4j\" (UniqueName: \"kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j\") pod \"maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.232623 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.232574 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894085-5znlw\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.232623 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.232616 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.232782 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.232765 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxf4\" (UniqueName: \"kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4\") pod \"maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.232838 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.232793 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9xr\" (UniqueName: \"kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr\") pod \"maas-api-key-cleanup-29894085-5znlw\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.233966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.233949 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.269075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.269041 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v"] Nov 02 18:45:00.274143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.274111 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw"] Nov 02 18:45:00.274349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.274231 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.274841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.274822 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8"] Nov 02 18:45:00.280425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.280399 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw"] Nov 02 18:45:00.322139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.322100 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw"] Nov 02 18:45:00.326543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.326508 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd"] Nov 02 18:45:00.334404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334370 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.334404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rz4j\" (UniqueName: \"kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j\") pod \"maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.334658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334438 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.334658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334467 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894085-5znlw\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.334658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334496 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.334658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334527 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.334658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbj4\" (UniqueName: \"kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.334658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334607 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.335054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334665 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhspz\" (UniqueName: \"kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.335054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334713 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.335054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.334814 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxf4\" (UniqueName: \"kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4\") pod \"maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.335245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.335083 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9xr\" (UniqueName: \"kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr\") pod \"maas-api-key-cleanup-29894085-5znlw\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.335245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.335160 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcb46\" (UniqueName: \"kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.335364 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.335268 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.335425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.335387 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59wm\" (UniqueName: \"kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.335490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.335476 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.336032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.336007 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894085-5znlw\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.371700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.371643 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65"] Nov 02 18:45:00.375272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.375246 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v"] Nov 02 18:45:00.392798 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.392766 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9xr\" (UniqueName: \"kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr\") pod \"maas-api-key-cleanup-29894085-5znlw\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.392960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.392831 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxf4\" (UniqueName: \"kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4\") pod \"maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.393247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.393227 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rz4j\" (UniqueName: \"kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j\") pod \"maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.436768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436641 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.436768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436687 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.437059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436778 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbj4\" (UniqueName: \"kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.437059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436843 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.437059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436894 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhspz\" (UniqueName: \"kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.437059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436940 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.437059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.436977 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcb46\" (UniqueName: \"kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.437059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.437006 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59wm\" (UniqueName: \"kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.437581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.437558 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.437682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.437604 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.437682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.437635 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.437839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.437819 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.485460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.485423 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:00.495213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.495184 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcb46\" (UniqueName: \"kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46\") pod \"maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.495528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.495500 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhspz\" (UniqueName: \"kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz\") pod \"maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.495795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.495773 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59wm\" (UniqueName: \"kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.496479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.496456 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbj4\" (UniqueName: \"kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.497299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.497280 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:00.506313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.506284 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:00.541320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.541293 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:00.549704 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.549663 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:00.556296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.555207 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:00.595493 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.589393 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:00.902880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.902844 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw"] Nov 02 18:45:00.903964 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:00.903936 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141e8f29_d446_4b6b_b03f_8b7c44589bb6.slice/crio-d50959a27755fb0cff564a44b9652e8b66bc5377862606aedbf090c6f6cc92b0 WatchSource:0}: Error finding container d50959a27755fb0cff564a44b9652e8b66bc5377862606aedbf090c6f6cc92b0: Status 404 returned error can't find the container with id d50959a27755fb0cff564a44b9652e8b66bc5377862606aedbf090c6f6cc92b0 Nov 02 18:45:00.948513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.948491 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8"] Nov 02 18:45:00.949125 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:00.949098 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc557152e_4d8b_4ef3_bada_524d1241dea9.slice/crio-8b3060366680564ae41aa4aeabd1341af0d1b603d4d827ee17696484f24d4737 WatchSource:0}: Error finding container 8b3060366680564ae41aa4aeabd1341af0d1b603d4d827ee17696484f24d4737: Status 404 returned error can't find the container with id 8b3060366680564ae41aa4aeabd1341af0d1b603d4d827ee17696484f24d4737 Nov 02 18:45:00.974995 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:00.974955 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw"] Nov 02 18:45:00.977021 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:00.976991 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20345a4_0e32_46ff_8ceb_9bef1696674f.slice/crio-47c68b0ab519551498fe6e47a19b674eb162ebbe514b5218ec6cb1d84e42a85e WatchSource:0}: Error finding container 47c68b0ab519551498fe6e47a19b674eb162ebbe514b5218ec6cb1d84e42a85e: Status 404 returned error can't find the container with id 47c68b0ab519551498fe6e47a19b674eb162ebbe514b5218ec6cb1d84e42a85e Nov 02 18:45:01.026911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.026883 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw"] Nov 02 18:45:01.038119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.038093 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65"] Nov 02 18:45:01.056795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.056767 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v"] Nov 02 18:45:01.105848 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:01.104895 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf514316_4a77_4f04_a159_44e445a49443.slice/crio-0e04337c08c3e1dddff932667af6ecf491e18f25d7e40329915fe40f3ce6d325 WatchSource:0}: Error finding container 0e04337c08c3e1dddff932667af6ecf491e18f25d7e40329915fe40f3ce6d325: Status 404 returned error can't find the container with id 0e04337c08c3e1dddff932667af6ecf491e18f25d7e40329915fe40f3ce6d325 Nov 02 18:45:01.106867 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:01.106828 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707ea234_0e7a_424f_9517_d2d7a64e98be.slice/crio-6270404c7a2a521cc5f966880be43d731124b479adca875ded3dea024ce6d91f WatchSource:0}: Error finding container 6270404c7a2a521cc5f966880be43d731124b479adca875ded3dea024ce6d91f: Status 404 returned error can't find the container with id 6270404c7a2a521cc5f966880be43d731124b479adca875ded3dea024ce6d91f Nov 02 18:45:01.107497 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:01.107420 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f26395e_1856_4992_acba_d8333eed6592.slice/crio-5424345cc52c6da03b87e70867a3dff0958c72daf0d12bf2c333df4e281bc4a5 WatchSource:0}: Error finding container 5424345cc52c6da03b87e70867a3dff0958c72daf0d12bf2c333df4e281bc4a5: Status 404 returned error can't find the container with id 5424345cc52c6da03b87e70867a3dff0958c72daf0d12bf2c333df4e281bc4a5 Nov 02 18:45:01.129868 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.129841 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd"] Nov 02 18:45:01.151152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.151108 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerStarted","Data":"6270404c7a2a521cc5f966880be43d731124b479adca875ded3dea024ce6d91f"} Nov 02 18:45:01.152654 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.152624 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerStarted","Data":"0e04337c08c3e1dddff932667af6ecf491e18f25d7e40329915fe40f3ce6d325"} Nov 02 18:45:01.153959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.153933 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerStarted","Data":"47c68b0ab519551498fe6e47a19b674eb162ebbe514b5218ec6cb1d84e42a85e"} Nov 02 18:45:01.155097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.155071 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerStarted","Data":"5424345cc52c6da03b87e70867a3dff0958c72daf0d12bf2c333df4e281bc4a5"} Nov 02 18:45:01.156304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.156278 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" event={"ID":"c557152e-4d8b-4ef3-bada-524d1241dea9","Type":"ContainerStarted","Data":"8b3060366680564ae41aa4aeabd1341af0d1b603d4d827ee17696484f24d4737"} Nov 02 18:45:01.157405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:01.157387 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" event={"ID":"141e8f29-d446-4b6b-b03f-8b7c44589bb6","Type":"ContainerStarted","Data":"d50959a27755fb0cff564a44b9652e8b66bc5377862606aedbf090c6f6cc92b0"} Nov 02 18:45:01.205436 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:01.205363 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd7c24d_7ec9_4945_867f_00f4536a94fe.slice/crio-3540b6a8368ba8b9685aad884edb4f2a2644a3bc05870594ccaf37aa440ac973 WatchSource:0}: Error finding container 3540b6a8368ba8b9685aad884edb4f2a2644a3bc05870594ccaf37aa440ac973: Status 404 returned error can't find the container with id 3540b6a8368ba8b9685aad884edb4f2a2644a3bc05870594ccaf37aa440ac973 Nov 02 18:45:02.165810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.165764 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerStarted","Data":"3403351d26e4c810f0bd816e068f85a8accc166b7361125c05c0de94c69bcb2b"} Nov 02 18:45:02.166027 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.165835 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerStarted","Data":"3540b6a8368ba8b9685aad884edb4f2a2644a3bc05870594ccaf37aa440ac973"} Nov 02 18:45:02.167466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.167438 5647 generic.go:356] "Generic (PLEG): container finished" podID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerID="65ee044902575c4daf04770d8ccb798a619d1b47b3d0146ece374387cfc2a0a9" exitCode=60 Nov 02 18:45:02.167598 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.167491 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" event={"ID":"c557152e-4d8b-4ef3-bada-524d1241dea9","Type":"ContainerDied","Data":"65ee044902575c4daf04770d8ccb798a619d1b47b3d0146ece374387cfc2a0a9"} Nov 02 18:45:02.167815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.167800 5647 scope.go:122] "RemoveContainer" containerID="65ee044902575c4daf04770d8ccb798a619d1b47b3d0146ece374387cfc2a0a9" Nov 02 18:45:02.169204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.169179 5647 generic.go:356] "Generic (PLEG): container finished" podID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerID="2e0964119fbd67910d88f9f9ec217df68ae87366fc1d65493c747af1ef694d6f" exitCode=60 Nov 02 18:45:02.169311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.169233 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" event={"ID":"141e8f29-d446-4b6b-b03f-8b7c44589bb6","Type":"ContainerDied","Data":"2e0964119fbd67910d88f9f9ec217df68ae87366fc1d65493c747af1ef694d6f"} Nov 02 18:45:02.169507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.169445 5647 scope.go:122] "RemoveContainer" containerID="2e0964119fbd67910d88f9f9ec217df68ae87366fc1d65493c747af1ef694d6f" Nov 02 18:45:02.171286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.171115 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerStarted","Data":"fc9a37e5d2e56271230c360fbd76104783cd22f3a7891151bb629b928541a367"} Nov 02 18:45:02.173080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.173041 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerStarted","Data":"040d6d455d8211190e24d49e015fc33c48d0d661c319844b4cb083105e10820b"} Nov 02 18:45:02.176537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.176367 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerStarted","Data":"0ea91bf21b2e67123c2b7f13042166197248ae8e5a392a28d54546c7189c42ca"} Nov 02 18:45:02.178799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.178769 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerStarted","Data":"8d4065ad7ed989566ab3c4d14051381bee40111c29e4d28148824f5f2e80bb1a"} Nov 02 18:45:02.285184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.285137 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" podStartSLOduration=2.285120259 podStartE2EDuration="2.285120259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.281726899 +0000 UTC m=+4889.355254101" watchObservedRunningTime="2026-11-02 18:45:02.285120259 +0000 UTC m=+4889.358647462" Nov 02 18:45:02.634209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.634156 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" podStartSLOduration=2.6341403740000002 podStartE2EDuration="2.634140374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.515035886 +0000 UTC m=+4889.588563090" watchObservedRunningTime="2026-11-02 18:45:02.634140374 +0000 UTC m=+4889.707667600" Nov 02 18:45:02.636494 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.636458 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" podStartSLOduration=2.636447505 podStartE2EDuration="2.636447505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.633496603 +0000 UTC m=+4889.707023805" watchObservedRunningTime="2026-11-02 18:45:02.636447505 +0000 UTC m=+4889.709974707" Nov 02 18:45:02.749810 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.749718 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" podStartSLOduration=2.749701371 podStartE2EDuration="2.749701371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.746982431 +0000 UTC m=+4889.820509635" watchObservedRunningTime="2026-11-02 18:45:02.749701371 +0000 UTC m=+4889.823228573" Nov 02 18:45:02.977974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:02.977865 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" podStartSLOduration=2.977850321 podStartE2EDuration="2.977850321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:02.976669901 +0000 UTC m=+4890.050197105" watchObservedRunningTime="2026-11-02 18:45:02.977850321 +0000 UTC m=+4890.051377524" Nov 02 18:45:03.186673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.186634 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f26395e-1856-4992-acba-d8333eed6592" containerID="8d4065ad7ed989566ab3c4d14051381bee40111c29e4d28148824f5f2e80bb1a" exitCode=60 Nov 02 18:45:03.186924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.186724 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerDied","Data":"8d4065ad7ed989566ab3c4d14051381bee40111c29e4d28148824f5f2e80bb1a"} Nov 02 18:45:03.187073 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.187059 5647 scope.go:122] "RemoveContainer" containerID="8d4065ad7ed989566ab3c4d14051381bee40111c29e4d28148824f5f2e80bb1a" Nov 02 18:45:03.188703 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.188681 5647 generic.go:356] "Generic (PLEG): container finished" podID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerID="3403351d26e4c810f0bd816e068f85a8accc166b7361125c05c0de94c69bcb2b" exitCode=60 Nov 02 18:45:03.189548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.188775 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerDied","Data":"3403351d26e4c810f0bd816e068f85a8accc166b7361125c05c0de94c69bcb2b"} Nov 02 18:45:03.189548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.189004 5647 scope.go:122] "RemoveContainer" containerID="3403351d26e4c810f0bd816e068f85a8accc166b7361125c05c0de94c69bcb2b" Nov 02 18:45:03.191196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.191175 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" event={"ID":"c557152e-4d8b-4ef3-bada-524d1241dea9","Type":"ContainerStarted","Data":"8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0"} Nov 02 18:45:03.193424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.193385 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" event={"ID":"141e8f29-d446-4b6b-b03f-8b7c44589bb6","Type":"ContainerStarted","Data":"80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d"} Nov 02 18:45:03.195043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.195018 5647 generic.go:356] "Generic (PLEG): container finished" podID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerID="fc9a37e5d2e56271230c360fbd76104783cd22f3a7891151bb629b928541a367" exitCode=60 Nov 02 18:45:03.195138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.195091 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerDied","Data":"fc9a37e5d2e56271230c360fbd76104783cd22f3a7891151bb629b928541a367"} Nov 02 18:45:03.195295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.195277 5647 scope.go:122] "RemoveContainer" containerID="fc9a37e5d2e56271230c360fbd76104783cd22f3a7891151bb629b928541a367" Nov 02 18:45:03.196965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.196941 5647 generic.go:356] "Generic (PLEG): container finished" podID="af514316-4a77-4f04-a159-44e445a49443" containerID="040d6d455d8211190e24d49e015fc33c48d0d661c319844b4cb083105e10820b" exitCode=60 Nov 02 18:45:03.197072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.196971 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerDied","Data":"040d6d455d8211190e24d49e015fc33c48d0d661c319844b4cb083105e10820b"} Nov 02 18:45:03.197245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.197228 5647 scope.go:122] "RemoveContainer" containerID="040d6d455d8211190e24d49e015fc33c48d0d661c319844b4cb083105e10820b" Nov 02 18:45:03.198650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.198619 5647 generic.go:356] "Generic (PLEG): container finished" podID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerID="0ea91bf21b2e67123c2b7f13042166197248ae8e5a392a28d54546c7189c42ca" exitCode=60 Nov 02 18:45:03.198777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.198653 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerDied","Data":"0ea91bf21b2e67123c2b7f13042166197248ae8e5a392a28d54546c7189c42ca"} Nov 02 18:45:03.198952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.198931 5647 scope.go:122] "RemoveContainer" containerID="0ea91bf21b2e67123c2b7f13042166197248ae8e5a392a28d54546c7189c42ca" Nov 02 18:45:03.769540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.769439 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" podStartSLOduration=3.76942571 podStartE2EDuration="3.76942571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:03.769185818 +0000 UTC m=+4890.842713020" watchObservedRunningTime="2026-11-02 18:45:03.76942571 +0000 UTC m=+4890.842952912" Nov 02 18:45:03.885561 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:03.885509 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" podStartSLOduration=3.885494361 podStartE2EDuration="3.885494361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:03.883172668 +0000 UTC m=+4890.956699870" watchObservedRunningTime="2026-11-02 18:45:03.885494361 +0000 UTC m=+4890.959021563" Nov 02 18:45:04.214328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.214287 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerStarted","Data":"e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006"} Nov 02 18:45:04.216508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.216474 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerStarted","Data":"fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344"} Nov 02 18:45:04.218494 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.218468 5647 generic.go:356] "Generic (PLEG): container finished" podID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerID="8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0" exitCode=60 Nov 02 18:45:04.218632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.218548 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" event={"ID":"c557152e-4d8b-4ef3-bada-524d1241dea9","Type":"ContainerDied","Data":"8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0"} Nov 02 18:45:04.218632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.218614 5647 scope.go:122] "RemoveContainer" containerID="65ee044902575c4daf04770d8ccb798a619d1b47b3d0146ece374387cfc2a0a9" Nov 02 18:45:04.218900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.218842 5647 scope.go:122] "RemoveContainer" containerID="8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0" Nov 02 18:45:04.219270 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:04.219172 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8_odh-ai-gateway-infra(c557152e-4d8b-4ef3-bada-524d1241dea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" Nov 02 18:45:04.220661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.220637 5647 generic.go:356] "Generic (PLEG): container finished" podID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerID="80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d" exitCode=60 Nov 02 18:45:04.220769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.220714 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" event={"ID":"141e8f29-d446-4b6b-b03f-8b7c44589bb6","Type":"ContainerDied","Data":"80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d"} Nov 02 18:45:04.220973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.220955 5647 scope.go:122] "RemoveContainer" containerID="80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d" Nov 02 18:45:04.221275 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:04.221252 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw_odh-ai-gateway-infra(141e8f29-d446-4b6b-b03f-8b7c44589bb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" Nov 02 18:45:04.222802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.222762 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerStarted","Data":"982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9"} Nov 02 18:45:04.223096 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.223076 5647 scope.go:122] "RemoveContainer" containerID="982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9" Nov 02 18:45:04.223420 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:04.223388 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65_odh-ai-gateway-infra(707ea234-0e7a-424f-9517-d2d7a64e98be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" Nov 02 18:45:04.225098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.225072 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerStarted","Data":"4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d"} Nov 02 18:45:04.227234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.227213 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerStarted","Data":"56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4"} Nov 02 18:45:04.283102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:04.283074 5647 scope.go:122] "RemoveContainer" containerID="2e0964119fbd67910d88f9f9ec217df68ae87366fc1d65493c747af1ef694d6f" Nov 02 18:45:05.053314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.053274 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp"] Nov 02 18:45:05.055877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.055847 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c1d6abf-wqhxp"] Nov 02 18:45:05.083336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.083298 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b20b57-e3f1-4069-ae2d-45b2a5957f54" path="/var/lib/kubelet/pods/30b20b57-e3f1-4069-ae2d-45b2a5957f54/volumes" Nov 02 18:45:05.235781 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.235754 5647 scope.go:122] "RemoveContainer" containerID="8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0" Nov 02 18:45:05.236135 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.236101 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8_odh-ai-gateway-infra(c557152e-4d8b-4ef3-bada-524d1241dea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" Nov 02 18:45:05.237621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.237592 5647 scope.go:122] "RemoveContainer" containerID="80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d" Nov 02 18:45:05.237955 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.237926 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw_odh-ai-gateway-infra(141e8f29-d446-4b6b-b03f-8b7c44589bb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" Nov 02 18:45:05.239497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.239465 5647 generic.go:356] "Generic (PLEG): container finished" podID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerID="982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9" exitCode=60 Nov 02 18:45:05.239624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.239504 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerDied","Data":"982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9"} Nov 02 18:45:05.239624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.239569 5647 scope.go:122] "RemoveContainer" containerID="fc9a37e5d2e56271230c360fbd76104783cd22f3a7891151bb629b928541a367" Nov 02 18:45:05.240105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.240085 5647 scope.go:122] "RemoveContainer" containerID="982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9" Nov 02 18:45:05.240378 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.240360 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65_odh-ai-gateway-infra(707ea234-0e7a-424f-9517-d2d7a64e98be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" Nov 02 18:45:05.241920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.241897 5647 generic.go:356] "Generic (PLEG): container finished" podID="af514316-4a77-4f04-a159-44e445a49443" containerID="4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d" exitCode=60 Nov 02 18:45:05.242041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.241974 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerDied","Data":"4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d"} Nov 02 18:45:05.242178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.242162 5647 scope.go:122] "RemoveContainer" containerID="4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d" Nov 02 18:45:05.242417 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.242394 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw_odh-ai-gateway-infra(af514316-4a77-4f04-a159-44e445a49443)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" podUID="af514316-4a77-4f04-a159-44e445a49443" Nov 02 18:45:05.243794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.243775 5647 generic.go:356] "Generic (PLEG): container finished" podID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerID="56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4" exitCode=60 Nov 02 18:45:05.243894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.243841 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerDied","Data":"56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4"} Nov 02 18:45:05.244121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.244109 5647 scope.go:122] "RemoveContainer" containerID="56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4" Nov 02 18:45:05.244360 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.244335 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894085-5znlw_odh-ai-gateway-infra(e20345a4-0e32-46ff-8ceb-9bef1696674f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" Nov 02 18:45:05.245685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.245659 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f26395e-1856-4992-acba-d8333eed6592" containerID="e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006" exitCode=60 Nov 02 18:45:05.245846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.245763 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerDied","Data":"e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006"} Nov 02 18:45:05.245941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.245921 5647 scope.go:122] "RemoveContainer" containerID="e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006" Nov 02 18:45:05.246159 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.246137 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v_odh-ai-gateway-infra(9f26395e-1856-4992-acba-d8333eed6592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" podUID="9f26395e-1856-4992-acba-d8333eed6592" Nov 02 18:45:05.247774 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.247751 5647 generic.go:356] "Generic (PLEG): container finished" podID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerID="fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344" exitCode=60 Nov 02 18:45:05.247876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.247800 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerDied","Data":"fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344"} Nov 02 18:45:05.248077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.248062 5647 scope.go:122] "RemoveContainer" containerID="fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344" Nov 02 18:45:05.248299 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.248280 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd_odh-ai-gateway-infra(0fd7c24d-7ec9-4945-867f-00f4536a94fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" Nov 02 18:45:05.255832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.255811 5647 scope.go:122] "RemoveContainer" containerID="040d6d455d8211190e24d49e015fc33c48d0d661c319844b4cb083105e10820b" Nov 02 18:45:05.269572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.269553 5647 scope.go:122] "RemoveContainer" containerID="0ea91bf21b2e67123c2b7f13042166197248ae8e5a392a28d54546c7189c42ca" Nov 02 18:45:05.284239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.284216 5647 scope.go:122] "RemoveContainer" containerID="8d4065ad7ed989566ab3c4d14051381bee40111c29e4d28148824f5f2e80bb1a" Nov 02 18:45:05.298263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:05.298240 5647 scope.go:122] "RemoveContainer" containerID="3403351d26e4c810f0bd816e068f85a8accc166b7361125c05c0de94c69bcb2b" Nov 02 18:45:05.949108 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:05.949068 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:45:06.257587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:06.257500 5647 scope.go:122] "RemoveContainer" containerID="fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344" Nov 02 18:45:06.258107 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:06.257902 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd_odh-ai-gateway-infra(0fd7c24d-7ec9-4945-867f-00f4536a94fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" Nov 02 18:45:06.259292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:06.259270 5647 scope.go:122] "RemoveContainer" containerID="982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9" Nov 02 18:45:06.259616 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:06.259586 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65_odh-ai-gateway-infra(707ea234-0e7a-424f-9517-d2d7a64e98be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" Nov 02 18:45:06.261044 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:06.261021 5647 scope.go:122] "RemoveContainer" containerID="4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d" Nov 02 18:45:06.261280 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:06.261262 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw_odh-ai-gateway-infra(af514316-4a77-4f04-a159-44e445a49443)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" podUID="af514316-4a77-4f04-a159-44e445a49443" Nov 02 18:45:06.262571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:06.262555 5647 scope.go:122] "RemoveContainer" containerID="56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4" Nov 02 18:45:06.262836 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:06.262808 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894085-5znlw_odh-ai-gateway-infra(e20345a4-0e32-46ff-8ceb-9bef1696674f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" Nov 02 18:45:06.264092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:06.264076 5647 scope.go:122] "RemoveContainer" containerID="e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006" Nov 02 18:45:06.264279 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:06.264263 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v_odh-ai-gateway-infra(9f26395e-1856-4992-acba-d8333eed6592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" podUID="9f26395e-1856-4992-acba-d8333eed6592" Nov 02 18:45:14.074826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:14.074787 5647 scope.go:122] "RemoveContainer" containerID="80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d" Nov 02 18:45:15.074554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:15.074523 5647 scope.go:122] "RemoveContainer" containerID="982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9" Nov 02 18:45:15.075108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:15.074686 5647 scope.go:122] "RemoveContainer" containerID="fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344" Nov 02 18:45:15.328388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:15.328357 5647 generic.go:356] "Generic (PLEG): container finished" podID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerID="a76713dd9a742cca0be4ad0b8501f2b974971930a8cdfc31142da05734b77ec2" exitCode=60 Nov 02 18:45:15.328565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:15.328432 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" event={"ID":"141e8f29-d446-4b6b-b03f-8b7c44589bb6","Type":"ContainerDied","Data":"a76713dd9a742cca0be4ad0b8501f2b974971930a8cdfc31142da05734b77ec2"} Nov 02 18:45:15.328565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:15.328503 5647 scope.go:122] "RemoveContainer" containerID="80abb7b755c73320462d1b684aa7ea3d5c35cae5caa135d59b39230173a8582d" Nov 02 18:45:15.328931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:15.328900 5647 scope.go:122] "RemoveContainer" containerID="a76713dd9a742cca0be4ad0b8501f2b974971930a8cdfc31142da05734b77ec2" Nov 02 18:45:15.329280 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:15.329261 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw_odh-ai-gateway-infra(141e8f29-d446-4b6b-b03f-8b7c44589bb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" Nov 02 18:45:16.336813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.336788 5647 generic.go:356] "Generic (PLEG): container finished" podID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerID="698ea6c981645f3d31ef4e44ba81d10ada54a1dd139d003a143b494c54337ce7" exitCode=60 Nov 02 18:45:16.337249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.336860 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerDied","Data":"698ea6c981645f3d31ef4e44ba81d10ada54a1dd139d003a143b494c54337ce7"} Nov 02 18:45:16.337249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.336920 5647 scope.go:122] "RemoveContainer" containerID="fde325e5ea55d8e311da7c9f758f490b0827146b9adf9c5c1bc62216d4847344" Nov 02 18:45:16.337404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.337367 5647 scope.go:122] "RemoveContainer" containerID="698ea6c981645f3d31ef4e44ba81d10ada54a1dd139d003a143b494c54337ce7" Nov 02 18:45:16.337809 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:16.337777 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd_odh-ai-gateway-infra(0fd7c24d-7ec9-4945-867f-00f4536a94fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" Nov 02 18:45:16.340605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.340582 5647 generic.go:356] "Generic (PLEG): container finished" podID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerID="91aba86ea4ebf48df9ea9997f2d6f68c430ef2d724555420acfec00b9af67938" exitCode=60 Nov 02 18:45:16.340718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.340660 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerDied","Data":"91aba86ea4ebf48df9ea9997f2d6f68c430ef2d724555420acfec00b9af67938"} Nov 02 18:45:16.341129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.341111 5647 scope.go:122] "RemoveContainer" containerID="91aba86ea4ebf48df9ea9997f2d6f68c430ef2d724555420acfec00b9af67938" Nov 02 18:45:16.341495 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:16.341473 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65_odh-ai-gateway-infra(707ea234-0e7a-424f-9517-d2d7a64e98be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" Nov 02 18:45:16.350152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.350132 5647 scope.go:122] "RemoveContainer" containerID="982d32fdae6935522c876a50b9842a09c8547b2829248b8d285851f4c2fbc2c9" Nov 02 18:45:16.505256 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.505225 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw"] Nov 02 18:45:16.671484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.671461 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:16.775067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.775031 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca\") pod \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " Nov 02 18:45:16.775260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.775137 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j\" (UniqueName: \"kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j\") pod \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\" (UID: \"141e8f29-d446-4b6b-b03f-8b7c44589bb6\") " Nov 02 18:45:16.775384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.775361 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca" pod "141e8f29-d446-4b6b-b03f-8b7c44589bb6" (UID: "141e8f29-d446-4b6b-b03f-8b7c44589bb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:16.775609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.775590 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/141e8f29-d446-4b6b-b03f-8b7c44589bb6-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:16.777586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.777566 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j" pod "141e8f29-d446-4b6b-b03f-8b7c44589bb6" (UID: "141e8f29-d446-4b6b-b03f-8b7c44589bb6"). InnerVolumeSpecName "kube-api-access-8rz4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:16.877165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:16.877129 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rz4j\" (UniqueName: \"kubernetes.io/projected/141e8f29-d446-4b6b-b03f-8b7c44589bb6-kube-api-access-8rz4j\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:17.074194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.074104 5647 scope.go:122] "RemoveContainer" containerID="8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0" Nov 02 18:45:17.074393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.074245 5647 scope.go:122] "RemoveContainer" containerID="56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4" Nov 02 18:45:17.352674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.352631 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" event={"ID":"141e8f29-d446-4b6b-b03f-8b7c44589bb6","Type":"ContainerDied","Data":"d50959a27755fb0cff564a44b9652e8b66bc5377862606aedbf090c6f6cc92b0"} Nov 02 18:45:17.352674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.352664 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw" Nov 02 18:45:17.353208 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.352705 5647 scope.go:122] "RemoveContainer" containerID="a76713dd9a742cca0be4ad0b8501f2b974971930a8cdfc31142da05734b77ec2" Nov 02 18:45:17.461554 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.461521 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd"] Nov 02 18:45:17.522097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.522062 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw"] Nov 02 18:45:17.525094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.525071 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fe24d-29894085-9q9bw"] Nov 02 18:45:17.580872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.580840 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65"] Nov 02 18:45:17.680874 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.678307 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:17.784105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.784081 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:17.788979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.788940 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz\" (UniqueName: \"kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz\") pod \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " Nov 02 18:45:17.789271 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.789231 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca\") pod \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\" (UID: \"0fd7c24d-7ec9-4945-867f-00f4536a94fe\") " Nov 02 18:45:17.789594 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.789573 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca" pod "0fd7c24d-7ec9-4945-867f-00f4536a94fe" (UID: "0fd7c24d-7ec9-4945-867f-00f4536a94fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:17.789714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.789696 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd7c24d-7ec9-4945-867f-00f4536a94fe-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:17.791272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.791253 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz" pod "0fd7c24d-7ec9-4945-867f-00f4536a94fe" (UID: "0fd7c24d-7ec9-4945-867f-00f4536a94fe"). InnerVolumeSpecName "kube-api-access-rhspz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:17.891444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.891359 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca\") pod \"707ea234-0e7a-424f-9517-d2d7a64e98be\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " Nov 02 18:45:17.891613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.891456 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm\" (UniqueName: \"kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm\") pod \"707ea234-0e7a-424f-9517-d2d7a64e98be\" (UID: \"707ea234-0e7a-424f-9517-d2d7a64e98be\") " Nov 02 18:45:17.891834 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.891809 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca" pod "707ea234-0e7a-424f-9517-d2d7a64e98be" (UID: "707ea234-0e7a-424f-9517-d2d7a64e98be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:17.892182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.892154 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhspz\" (UniqueName: \"kubernetes.io/projected/0fd7c24d-7ec9-4945-867f-00f4536a94fe-kube-api-access-rhspz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:17.892182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.892184 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707ea234-0e7a-424f-9517-d2d7a64e98be-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:17.893814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.893782 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm" pod "707ea234-0e7a-424f-9517-d2d7a64e98be" (UID: "707ea234-0e7a-424f-9517-d2d7a64e98be"). InnerVolumeSpecName "kube-api-access-n59wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:17.993415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:17.993372 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n59wm\" (UniqueName: \"kubernetes.io/projected/707ea234-0e7a-424f-9517-d2d7a64e98be-kube-api-access-n59wm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:18.074172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.074139 5647 scope.go:122] "RemoveContainer" containerID="e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006" Nov 02 18:45:18.074358 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.074209 5647 scope.go:122] "RemoveContainer" containerID="4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d" Nov 02 18:45:18.361240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.361206 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" Nov 02 18:45:18.361700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.361204 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd" event={"ID":"0fd7c24d-7ec9-4945-867f-00f4536a94fe","Type":"ContainerDied","Data":"3540b6a8368ba8b9685aad884edb4f2a2644a3bc05870594ccaf37aa440ac973"} Nov 02 18:45:18.361700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.361360 5647 scope.go:122] "RemoveContainer" containerID="698ea6c981645f3d31ef4e44ba81d10ada54a1dd139d003a143b494c54337ce7" Nov 02 18:45:18.363612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.363585 5647 generic.go:356] "Generic (PLEG): container finished" podID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerID="6c15526ce67b74e95749a57fd6b68361c09ecb3e99fbf25f57d7929afc1085f7" exitCode=60 Nov 02 18:45:18.363847 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.363650 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" event={"ID":"c557152e-4d8b-4ef3-bada-524d1241dea9","Type":"ContainerDied","Data":"6c15526ce67b74e95749a57fd6b68361c09ecb3e99fbf25f57d7929afc1085f7"} Nov 02 18:45:18.364084 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.364067 5647 scope.go:122] "RemoveContainer" containerID="6c15526ce67b74e95749a57fd6b68361c09ecb3e99fbf25f57d7929afc1085f7" Nov 02 18:45:18.364382 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:18.364356 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8_odh-ai-gateway-infra(c557152e-4d8b-4ef3-bada-524d1241dea9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" Nov 02 18:45:18.366349 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.366316 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" Nov 02 18:45:18.366475 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.366353 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65" event={"ID":"707ea234-0e7a-424f-9517-d2d7a64e98be","Type":"ContainerDied","Data":"6270404c7a2a521cc5f966880be43d731124b479adca875ded3dea024ce6d91f"} Nov 02 18:45:18.368317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.368293 5647 generic.go:356] "Generic (PLEG): container finished" podID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerID="eadcf25f6dd6055dcaa363d5d15c659121c4575416903b1c083add3b60405a32" exitCode=60 Nov 02 18:45:18.368425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.368327 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerDied","Data":"eadcf25f6dd6055dcaa363d5d15c659121c4575416903b1c083add3b60405a32"} Nov 02 18:45:18.368778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.368723 5647 scope.go:122] "RemoveContainer" containerID="eadcf25f6dd6055dcaa363d5d15c659121c4575416903b1c083add3b60405a32" Nov 02 18:45:18.369071 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:18.369052 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894085-5znlw_odh-ai-gateway-infra(e20345a4-0e32-46ff-8ceb-9bef1696674f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" Nov 02 18:45:18.372901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.372884 5647 scope.go:122] "RemoveContainer" containerID="8ebde97fa479235571da401fc96c421d05bffa138db70d0c223b5b44b27a30c0" Nov 02 18:45:18.387259 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.387234 5647 scope.go:122] "RemoveContainer" containerID="91aba86ea4ebf48df9ea9997f2d6f68c430ef2d724555420acfec00b9af67938" Nov 02 18:45:18.396991 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.396970 5647 scope.go:122] "RemoveContainer" containerID="56b30186506739e87f464a8273f90f9afbfc5b103d070146db2e43128ac6b5b4" Nov 02 18:45:18.644484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.644452 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65"] Nov 02 18:45:18.647308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.647273 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-dd92ac-29894085-ckg65"] Nov 02 18:45:18.870306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.870270 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd"] Nov 02 18:45:18.874600 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:18.874566 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ceb1ae-29894085-rwqdd"] Nov 02 18:45:19.082344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.082309 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" path="/var/lib/kubelet/pods/0fd7c24d-7ec9-4945-867f-00f4536a94fe/volumes" Nov 02 18:45:19.082838 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.082820 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" path="/var/lib/kubelet/pods/141e8f29-d446-4b6b-b03f-8b7c44589bb6/volumes" Nov 02 18:45:19.083442 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.083419 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" path="/var/lib/kubelet/pods/707ea234-0e7a-424f-9517-d2d7a64e98be/volumes" Nov 02 18:45:19.380263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.380169 5647 generic.go:356] "Generic (PLEG): container finished" podID="af514316-4a77-4f04-a159-44e445a49443" containerID="f6f66336a33e7b6614dc46ac9908a577288dac37694cfec2de9cd7a549d61c5c" exitCode=60 Nov 02 18:45:19.380263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.380244 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerDied","Data":"f6f66336a33e7b6614dc46ac9908a577288dac37694cfec2de9cd7a549d61c5c"} Nov 02 18:45:19.380814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.380316 5647 scope.go:122] "RemoveContainer" containerID="4c515fdb9f96d5ca9c9af694eceecdc317b11d1ade6af673ba1bc6fe295e5a5d" Nov 02 18:45:19.380814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.380802 5647 scope.go:122] "RemoveContainer" containerID="f6f66336a33e7b6614dc46ac9908a577288dac37694cfec2de9cd7a549d61c5c" Nov 02 18:45:19.381245 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:19.381217 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw_odh-ai-gateway-infra(af514316-4a77-4f04-a159-44e445a49443)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" podUID="af514316-4a77-4f04-a159-44e445a49443" Nov 02 18:45:19.384400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.384376 5647 generic.go:356] "Generic (PLEG): container finished" podID="9f26395e-1856-4992-acba-d8333eed6592" containerID="3ebcdff50cd2198cbaeb958c17a807b425093ed12f8dd42d06005677ce6d5f36" exitCode=60 Nov 02 18:45:19.384531 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.384458 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerDied","Data":"3ebcdff50cd2198cbaeb958c17a807b425093ed12f8dd42d06005677ce6d5f36"} Nov 02 18:45:19.384955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.384931 5647 scope.go:122] "RemoveContainer" containerID="3ebcdff50cd2198cbaeb958c17a807b425093ed12f8dd42d06005677ce6d5f36" Nov 02 18:45:19.385341 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:19.385284 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v_odh-ai-gateway-infra(9f26395e-1856-4992-acba-d8333eed6592)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" podUID="9f26395e-1856-4992-acba-d8333eed6592" Nov 02 18:45:19.393653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.393631 5647 scope.go:122] "RemoveContainer" containerID="e68b28906a28ad9920d10b20bff9758e32e6ce24ab4b08373bb8d97da844f006" Nov 02 18:45:19.540472 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.540439 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8"] Nov 02 18:45:19.704300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.704274 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:19.716313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.716285 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca\") pod \"c557152e-4d8b-4ef3-bada-524d1241dea9\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " Nov 02 18:45:19.716541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.716519 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4\" (UniqueName: \"kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4\") pod \"c557152e-4d8b-4ef3-bada-524d1241dea9\" (UID: \"c557152e-4d8b-4ef3-bada-524d1241dea9\") " Nov 02 18:45:19.716699 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.716677 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca" pod "c557152e-4d8b-4ef3-bada-524d1241dea9" (UID: "c557152e-4d8b-4ef3-bada-524d1241dea9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:19.716997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.716982 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c557152e-4d8b-4ef3-bada-524d1241dea9-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:19.718759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.718713 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4" pod "c557152e-4d8b-4ef3-bada-524d1241dea9" (UID: "c557152e-4d8b-4ef3-bada-524d1241dea9"). InnerVolumeSpecName "kube-api-access-jmxf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:19.770181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.770148 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw"] Nov 02 18:45:19.818300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.818263 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxf4\" (UniqueName: \"kubernetes.io/projected/c557152e-4d8b-4ef3-bada-524d1241dea9-kube-api-access-jmxf4\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:19.931134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:19.931056 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:20.020505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.020440 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr\" (UniqueName: \"kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr\") pod \"e20345a4-0e32-46ff-8ceb-9bef1696674f\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " Nov 02 18:45:20.020953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.020693 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca\") pod \"e20345a4-0e32-46ff-8ceb-9bef1696674f\" (UID: \"e20345a4-0e32-46ff-8ceb-9bef1696674f\") " Nov 02 18:45:20.021065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.021040 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca" pod "e20345a4-0e32-46ff-8ceb-9bef1696674f" (UID: "e20345a4-0e32-46ff-8ceb-9bef1696674f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:20.021213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.021199 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e20345a4-0e32-46ff-8ceb-9bef1696674f-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:20.022991 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.022969 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr" pod "e20345a4-0e32-46ff-8ceb-9bef1696674f" (UID: "e20345a4-0e32-46ff-8ceb-9bef1696674f"). InnerVolumeSpecName "kube-api-access-4b9xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:20.125860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.124621 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9xr\" (UniqueName: \"kubernetes.io/projected/e20345a4-0e32-46ff-8ceb-9bef1696674f-kube-api-access-4b9xr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:20.398951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.398920 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" Nov 02 18:45:20.399380 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.398946 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8" event={"ID":"c557152e-4d8b-4ef3-bada-524d1241dea9","Type":"ContainerDied","Data":"8b3060366680564ae41aa4aeabd1341af0d1b603d4d827ee17696484f24d4737"} Nov 02 18:45:20.399380 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.399031 5647 scope.go:122] "RemoveContainer" containerID="6c15526ce67b74e95749a57fd6b68361c09ecb3e99fbf25f57d7929afc1085f7" Nov 02 18:45:20.403504 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.403475 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" Nov 02 18:45:20.403890 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.403474 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw" event={"ID":"e20345a4-0e32-46ff-8ceb-9bef1696674f","Type":"ContainerDied","Data":"47c68b0ab519551498fe6e47a19b674eb162ebbe514b5218ec6cb1d84e42a85e"} Nov 02 18:45:20.411850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.411825 5647 scope.go:122] "RemoveContainer" containerID="eadcf25f6dd6055dcaa363d5d15c659121c4575416903b1c083add3b60405a32" Nov 02 18:45:20.505621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.505590 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw"] Nov 02 18:45:20.567449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.567415 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw"] Nov 02 18:45:20.581459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.581427 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894085-5znlw"] Nov 02 18:45:20.620176 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.620146 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v"] Nov 02 18:45:20.696577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.696537 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8"] Nov 02 18:45:20.703152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.703121 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8f40154e-29894085-4fts8"] Nov 02 18:45:20.714245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.714213 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:20.732169 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.732132 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca\") pod \"af514316-4a77-4f04-a159-44e445a49443\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " Nov 02 18:45:20.732360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.732265 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4\" (UniqueName: \"kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4\") pod \"af514316-4a77-4f04-a159-44e445a49443\" (UID: \"af514316-4a77-4f04-a159-44e445a49443\") " Nov 02 18:45:20.732867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.732837 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca" pod "af514316-4a77-4f04-a159-44e445a49443" (UID: "af514316-4a77-4f04-a159-44e445a49443"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:20.735052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.735021 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4" pod "af514316-4a77-4f04-a159-44e445a49443" (UID: "af514316-4a77-4f04-a159-44e445a49443"). InnerVolumeSpecName "kube-api-access-8fbj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:20.796286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.796261 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:20.833460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.833425 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca\") pod \"9f26395e-1856-4992-acba-d8333eed6592\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " Nov 02 18:45:20.833757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.833716 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46\" (UniqueName: \"kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46\") pod \"9f26395e-1856-4992-acba-d8333eed6592\" (UID: \"9f26395e-1856-4992-acba-d8333eed6592\") " Nov 02 18:45:20.833887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.833865 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca" pod "9f26395e-1856-4992-acba-d8333eed6592" (UID: "9f26395e-1856-4992-acba-d8333eed6592"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:20.834171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.834156 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbj4\" (UniqueName: \"kubernetes.io/projected/af514316-4a77-4f04-a159-44e445a49443-kube-api-access-8fbj4\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:20.834255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.834173 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f26395e-1856-4992-acba-d8333eed6592-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:20.834255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.834183 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af514316-4a77-4f04-a159-44e445a49443-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:20.835972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.835944 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46" pod "9f26395e-1856-4992-acba-d8333eed6592" (UID: "9f26395e-1856-4992-acba-d8333eed6592"). InnerVolumeSpecName "kube-api-access-fcb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:20.935285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:20.935200 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcb46\" (UniqueName: \"kubernetes.io/projected/9f26395e-1856-4992-acba-d8333eed6592-kube-api-access-fcb46\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:21.085666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.085613 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" path="/var/lib/kubelet/pods/c557152e-4d8b-4ef3-bada-524d1241dea9/volumes" Nov 02 18:45:21.086147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.086129 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" path="/var/lib/kubelet/pods/e20345a4-0e32-46ff-8ceb-9bef1696674f/volumes" Nov 02 18:45:21.414319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.414287 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" Nov 02 18:45:21.414848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.414286 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw" event={"ID":"af514316-4a77-4f04-a159-44e445a49443","Type":"ContainerDied","Data":"0e04337c08c3e1dddff932667af6ecf491e18f25d7e40329915fe40f3ce6d325"} Nov 02 18:45:21.414848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.414416 5647 scope.go:122] "RemoveContainer" containerID="f6f66336a33e7b6614dc46ac9908a577288dac37694cfec2de9cd7a549d61c5c" Nov 02 18:45:21.416934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.416906 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" event={"ID":"9f26395e-1856-4992-acba-d8333eed6592","Type":"ContainerDied","Data":"5424345cc52c6da03b87e70867a3dff0958c72daf0d12bf2c333df4e281bc4a5"} Nov 02 18:45:21.416934 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.416922 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v" Nov 02 18:45:21.424701 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.424685 5647 scope.go:122] "RemoveContainer" containerID="3ebcdff50cd2198cbaeb958c17a807b425093ed12f8dd42d06005677ce6d5f36" Nov 02 18:45:21.583371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.583332 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v"] Nov 02 18:45:21.587062 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.587035 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-6e3175-29894085-2zv7v"] Nov 02 18:45:21.697119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.697018 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw"] Nov 02 18:45:21.701479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:21.701450 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-4ecad8-29894085-dh8qw"] Nov 02 18:45:22.009780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.009667 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv"] Nov 02 18:45:22.010313 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010296 5647 cpu_manager.go:414] "containerMap: removing container" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010315 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010326 5647 cpu_manager.go:414] "containerMap: removing container" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010332 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010340 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010344 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010354 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010359 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010369 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:22.010371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010373 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010380 5647 cpu_manager.go:414] "containerMap: removing container" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010385 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010400 5647 cpu_manager.go:414] "containerMap: removing container" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010404 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010411 5647 cpu_manager.go:414] "containerMap: removing container" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010415 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010424 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010429 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010436 5647 cpu_manager.go:414] "containerMap: removing container" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010441 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010446 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010450 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010465 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010469 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010481 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010485 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010499 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010504 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010509 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010512 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010612 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010619 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010628 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010633 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010639 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010644 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010649 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010655 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010659 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010664 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010670 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010677 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:22.010668 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010683 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:22.011783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010689 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:22.011783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.010695 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:22.015402 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.015385 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.049832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.049806 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.049972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.049843 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.050020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.050004 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khdsq\" (UniqueName: \"kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.068489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.068359 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:45:22.068489 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.068461 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:45:22.069440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.069415 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv"] Nov 02 18:45:22.152014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.151987 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.152181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.152024 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.152269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.152197 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khdsq\" (UniqueName: \"kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.152853 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.152830 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.154670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.154645 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.210175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.210145 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khdsq\" (UniqueName: \"kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq\") pod \"maas-api-revoke-sub-dec28c4f-rmzfv\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.328803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.328713 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:22.703098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:22.703069 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv"] Nov 02 18:45:22.703797 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:22.703773 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c41009_5028_4ea6_bd5d_85fbcb95cbe1.slice/crio-bf610e6a9541cf83dc17098e651d37ef9b4d6af0c3a45b4f91c7e48c5ff55f52 WatchSource:0}: Error finding container bf610e6a9541cf83dc17098e651d37ef9b4d6af0c3a45b4f91c7e48c5ff55f52: Status 404 returned error can't find the container with id bf610e6a9541cf83dc17098e651d37ef9b4d6af0c3a45b4f91c7e48c5ff55f52 Nov 02 18:45:23.082110 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:23.082078 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f26395e-1856-4992-acba-d8333eed6592" path="/var/lib/kubelet/pods/9f26395e-1856-4992-acba-d8333eed6592/volumes" Nov 02 18:45:23.082566 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:23.082551 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af514316-4a77-4f04-a159-44e445a49443" path="/var/lib/kubelet/pods/af514316-4a77-4f04-a159-44e445a49443/volumes" Nov 02 18:45:23.439328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:23.439294 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" event={"ID":"34c41009-5028-4ea6-bd5d-85fbcb95cbe1","Type":"ContainerStarted","Data":"be3fb045b122b95b78bfc7657eb7d6a987ba870a1ee2656ab891cf8577c70243"} Nov 02 18:45:23.439520 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:23.439343 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" event={"ID":"34c41009-5028-4ea6-bd5d-85fbcb95cbe1","Type":"ContainerStarted","Data":"bf610e6a9541cf83dc17098e651d37ef9b4d6af0c3a45b4f91c7e48c5ff55f52"} Nov 02 18:45:23.554508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:23.554453 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" podStartSLOduration=2.55443545 podStartE2EDuration="2.55443545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:23.553184537 +0000 UTC m=+4910.626711741" watchObservedRunningTime="2026-11-02 18:45:23.55443545 +0000 UTC m=+4910.627962841" Nov 02 18:45:24.447872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:24.447820 5647 generic.go:356] "Generic (PLEG): container finished" podID="34c41009-5028-4ea6-bd5d-85fbcb95cbe1" containerID="be3fb045b122b95b78bfc7657eb7d6a987ba870a1ee2656ab891cf8577c70243" exitCode=0 Nov 02 18:45:24.448355 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:24.447903 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" event={"ID":"34c41009-5028-4ea6-bd5d-85fbcb95cbe1","Type":"ContainerDied","Data":"be3fb045b122b95b78bfc7657eb7d6a987ba870a1ee2656ab891cf8577c70243"} Nov 02 18:45:25.617694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.617664 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:25.700108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.700075 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca\") pod \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " Nov 02 18:45:25.700288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.700139 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca\") pod \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " Nov 02 18:45:25.700382 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.700320 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq\" (UniqueName: \"kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq\") pod \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\" (UID: \"34c41009-5028-4ea6-bd5d-85fbcb95cbe1\") " Nov 02 18:45:25.700486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.700463 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca" pod "34c41009-5028-4ea6-bd5d-85fbcb95cbe1" (UID: "34c41009-5028-4ea6-bd5d-85fbcb95cbe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:25.700716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.700702 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:25.702484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.702461 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca" pod "34c41009-5028-4ea6-bd5d-85fbcb95cbe1" (UID: "34c41009-5028-4ea6-bd5d-85fbcb95cbe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:45:25.702586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.702502 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq" pod "34c41009-5028-4ea6-bd5d-85fbcb95cbe1" (UID: "34c41009-5028-4ea6-bd5d-85fbcb95cbe1"). InnerVolumeSpecName "kube-api-access-khdsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:25.802322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.802278 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khdsq\" (UniqueName: \"kubernetes.io/projected/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-kube-api-access-khdsq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:25.802322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:25.802316 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c41009-5028-4ea6-bd5d-85fbcb95cbe1-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:26.463442 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:26.463410 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" Nov 02 18:45:26.463647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:26.463457 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv" event={"ID":"34c41009-5028-4ea6-bd5d-85fbcb95cbe1","Type":"ContainerDied","Data":"bf610e6a9541cf83dc17098e651d37ef9b4d6af0c3a45b4f91c7e48c5ff55f52"} Nov 02 18:45:26.463647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:26.463512 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf610e6a9541cf83dc17098e651d37ef9b4d6af0c3a45b4f91c7e48c5ff55f52" Nov 02 18:45:35.009583 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.009542 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4"] Nov 02 18:45:35.010191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010176 5647 cpu_manager.go:414] "containerMap: removing container" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:35.010236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010192 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:35.010236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010207 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:35.010236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010212 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:35.010236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010224 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:35.010236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010229 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:35.010236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010235 5647 cpu_manager.go:414] "containerMap: removing container" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010239 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af514316-4a77-4f04-a159-44e445a49443" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010260 5647 cpu_manager.go:414] "containerMap: removing container" podUID="34c41009-5028-4ea6-bd5d-85fbcb95cbe1" containerName="revoke-keys" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010265 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c41009-5028-4ea6-bd5d-85fbcb95cbe1" containerName="revoke-keys" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010273 5647 cpu_manager.go:414] "containerMap: removing container" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010278 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010291 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010295 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010398 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="707ea234-0e7a-424f-9517-d2d7a64e98be" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010407 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c41009-5028-4ea6-bd5d-85fbcb95cbe1" containerName="revoke-keys" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010416 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c557152e-4d8b-4ef3-bada-524d1241dea9" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010425 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd7c24d-7ec9-4945-867f-00f4536a94fe" containerName="cleanup" Nov 02 18:45:35.010436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010433 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e20345a4-0e32-46ff-8ceb-9bef1696674f" containerName="cleanup" Nov 02 18:45:35.010844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010442 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f26395e-1856-4992-acba-d8333eed6592" containerName="cleanup" Nov 02 18:45:35.010844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.010450 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="141e8f29-d446-4b6b-b03f-8b7c44589bb6" containerName="cleanup" Nov 02 18:45:35.015630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.015608 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.069767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.069717 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4"] Nov 02 18:45:35.116820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.116786 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:45:35.116981 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.116832 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:45:35.136914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.135533 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.136914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.135600 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2knt\" (UniqueName: \"kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.136914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.135708 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.237444 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.237404 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.237636 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.237592 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.237686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.237635 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2knt\" (UniqueName: \"kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.238372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.238342 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.240021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.240000 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.295257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.295184 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2knt\" (UniqueName: \"kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt\") pod \"maas-api-revoke-sub-c74266d4-dgws4\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.329263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.329219 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:35.708155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:35.708127 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4"] Nov 02 18:45:35.709232 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:45:35.709203 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a26bf4_531b_4a9f_a586_bb0e1f39a756.slice/crio-bd756c0b1f57fb0f78a8de7f39570bf6e8d0c877f526bae57720ec5fc9534cea WatchSource:0}: Error finding container bd756c0b1f57fb0f78a8de7f39570bf6e8d0c877f526bae57720ec5fc9534cea: Status 404 returned error can't find the container with id bd756c0b1f57fb0f78a8de7f39570bf6e8d0c877f526bae57720ec5fc9534cea Nov 02 18:45:35.947504 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:45:35.947453 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:45:36.534055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:36.534018 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" event={"ID":"d4a26bf4-531b-4a9f-a586-bb0e1f39a756","Type":"ContainerStarted","Data":"bf10623808b00e64dc8dacc6855e0cd8ec0dfd7a3b6cd57357a4bcca583c0344"} Nov 02 18:45:36.534477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:36.534071 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" event={"ID":"d4a26bf4-531b-4a9f-a586-bb0e1f39a756","Type":"ContainerStarted","Data":"bd756c0b1f57fb0f78a8de7f39570bf6e8d0c877f526bae57720ec5fc9534cea"} Nov 02 18:45:36.649053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:36.648981 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" podStartSLOduration=2.648966702 podStartE2EDuration="2.648966702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:45:36.648115531 +0000 UTC m=+4923.721642736" watchObservedRunningTime="2026-11-02 18:45:36.648966702 +0000 UTC m=+4923.722493905" Nov 02 18:45:37.544000 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:37.543961 5647 generic.go:356] "Generic (PLEG): container finished" podID="d4a26bf4-531b-4a9f-a586-bb0e1f39a756" containerID="bf10623808b00e64dc8dacc6855e0cd8ec0dfd7a3b6cd57357a4bcca583c0344" exitCode=0 Nov 02 18:45:37.544385 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:37.544103 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" event={"ID":"d4a26bf4-531b-4a9f-a586-bb0e1f39a756","Type":"ContainerDied","Data":"bf10623808b00e64dc8dacc6855e0cd8ec0dfd7a3b6cd57357a4bcca583c0344"} Nov 02 18:45:38.707016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.706983 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:38.785103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.785065 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt\" (UniqueName: \"kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt\") pod \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " Nov 02 18:45:38.785299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.785134 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca\") pod \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " Nov 02 18:45:38.785299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.785281 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca\") pod \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\" (UID: \"d4a26bf4-531b-4a9f-a586-bb0e1f39a756\") " Nov 02 18:45:38.785624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.785596 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca" pod "d4a26bf4-531b-4a9f-a586-bb0e1f39a756" (UID: "d4a26bf4-531b-4a9f-a586-bb0e1f39a756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:45:38.785839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.785814 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:38.787374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.787350 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca" pod "d4a26bf4-531b-4a9f-a586-bb0e1f39a756" (UID: "d4a26bf4-531b-4a9f-a586-bb0e1f39a756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:45:38.787374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.787365 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt" pod "d4a26bf4-531b-4a9f-a586-bb0e1f39a756" (UID: "d4a26bf4-531b-4a9f-a586-bb0e1f39a756"). InnerVolumeSpecName "kube-api-access-n2knt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:45:38.887615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.887572 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2knt\" (UniqueName: \"kubernetes.io/projected/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-kube-api-access-n2knt\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:38.887615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:38.887619 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4a26bf4-531b-4a9f-a586-bb0e1f39a756-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:45:39.558573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:39.558473 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" event={"ID":"d4a26bf4-531b-4a9f-a586-bb0e1f39a756","Type":"ContainerDied","Data":"bd756c0b1f57fb0f78a8de7f39570bf6e8d0c877f526bae57720ec5fc9534cea"} Nov 02 18:45:39.558573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:39.558500 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4" Nov 02 18:45:39.558573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:39.558553 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd756c0b1f57fb0f78a8de7f39570bf6e8d0c877f526bae57720ec5fc9534cea" Nov 02 18:45:49.904504 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:45:49.904466 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:46:05.948970 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:46:05.948923 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:46:10.237837 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:10.237794 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx"] Nov 02 18:46:10.349678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:10.349646 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn"] Nov 02 18:46:15.238884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.238798 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" podUID="2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" containerName="payload-processing" containerID="cri-o://8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd" gracePeriod=25 Nov 02 18:46:15.350825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.350782 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" podUID="32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" containerName="payload-pre-processing" containerID="cri-o://9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e" gracePeriod=25 Nov 02 18:46:15.512040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.512014 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:46:15.609843 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.609808 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume\") pod \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " Nov 02 18:46:15.610036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.609881 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh\" (UniqueName: \"kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh\") pod \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\" (UID: \"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf\") " Nov 02 18:46:15.610206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.610183 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume" pod "2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" (UID: "2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:15.610504 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.610488 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:15.612968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.612938 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh" pod "2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" (UID: "2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf"). InnerVolumeSpecName "kube-api-access-8ftlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:15.637428 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.637392 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:46:15.710923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.710888 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume\") pod \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " Nov 02 18:46:15.711129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.711058 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x\" (UniqueName: \"kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x\") pod \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\" (UID: \"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6\") " Nov 02 18:46:15.711326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.711293 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume" pod "32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" (UID: "32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:15.711670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.711637 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ftlh\" (UniqueName: \"kubernetes.io/projected/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf-kube-api-access-8ftlh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:15.711670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.711663 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:15.713610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.713588 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x" pod "32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" (UID: "32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6"). InnerVolumeSpecName "kube-api-access-ppt6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:15.805682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.805586 5647 generic.go:356] "Generic (PLEG): container finished" podID="32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" containerID="9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e" exitCode=0 Nov 02 18:46:15.805682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.805641 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" Nov 02 18:46:15.805682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.805672 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" event={"ID":"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6","Type":"ContainerDied","Data":"9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e"} Nov 02 18:46:15.806034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.805765 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn" event={"ID":"32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6","Type":"ContainerDied","Data":"f0549694ac60f7fc0b735ab9de2547e46883a4c227fdd55f053c615556281ee6"} Nov 02 18:46:15.806034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.805771 5647 scope.go:122] "RemoveContainer" containerID="9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e" Nov 02 18:46:15.807350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.807324 5647 generic.go:356] "Generic (PLEG): container finished" podID="2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" containerID="8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd" exitCode=0 Nov 02 18:46:15.807454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.807369 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" event={"ID":"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf","Type":"ContainerDied","Data":"8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd"} Nov 02 18:46:15.807454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.807411 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" event={"ID":"2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf","Type":"ContainerDied","Data":"b854e3eb8ce46ad41e2a10d527c58f9f735d1d1f00e65237e6f62bb1919b13bb"} Nov 02 18:46:15.807586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.807481 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx" Nov 02 18:46:15.812409 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.812384 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppt6x\" (UniqueName: \"kubernetes.io/projected/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6-kube-api-access-ppt6x\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:15.817229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.817211 5647 scope.go:122] "RemoveContainer" containerID="9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e" Nov 02 18:46:15.817584 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:46:15.817550 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e\": container with ID starting with 9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e not found: ID does not exist" containerID="9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e" Nov 02 18:46:15.817659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.817589 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e"} err="failed to get container status \"9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e\": rpc error: code = NotFound desc = could not find container \"9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e\": container with ID starting with 9e04cf015cec3249faa1fb96ebd157862ff5103e830632fe84ed64804f3f980e not found: ID does not exist" Nov 02 18:46:15.817659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.817609 5647 scope.go:122] "RemoveContainer" containerID="8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd" Nov 02 18:46:15.830511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.830492 5647 scope.go:122] "RemoveContainer" containerID="8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd" Nov 02 18:46:15.830857 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:46:15.830832 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd\": container with ID starting with 8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd not found: ID does not exist" containerID="8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd" Nov 02 18:46:15.830922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.830865 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd"} err="failed to get container status \"8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd\": rpc error: code = NotFound desc = could not find container \"8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd\": container with ID starting with 8edbdf3fb3bd316962004500e4fd8479840b335ebff0e4e348695778252ee7cd not found: ID does not exist" Nov 02 18:46:15.972463 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.972392 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx"] Nov 02 18:46:15.975998 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:15.975975 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8f40154e-56f7694565-fndwx"] Nov 02 18:46:16.086764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:16.086716 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn"] Nov 02 18:46:16.089580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:16.089557 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8f40154e-6bbc695755-fh4kn"] Nov 02 18:46:17.082015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:17.081984 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" path="/var/lib/kubelet/pods/2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf/volumes" Nov 02 18:46:17.082422 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:17.082390 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" path="/var/lib/kubelet/pods/32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6/volumes" Nov 02 18:46:19.488968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:19.488933 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc"] Nov 02 18:46:19.489426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:19.489152 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" containerID="cri-o://0b76941a7c0fd94c23156d8c9018b479a92fc2d84cc6825ccebf214458578e11" gracePeriod=30 Nov 02 18:46:19.539980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:19.539949 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb"] Nov 02 18:46:19.540179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:19.540162 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" containerID="cri-o://ca296b09f6a8eaf937438a68356d52dedffb8e98a6c189dd9c4110b0956e6e4a" gracePeriod=30 Nov 02 18:46:19.542723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:19.542699 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb"] Nov 02 18:46:19.542931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:19.542905 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" containerID="cri-o://06534d0ce220bbaa02c52f8d17e4bda7998c1bf7ce0e3f092f3302fb4c708737" gracePeriod=30 Nov 02 18:46:21.805190 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:21.805143 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb"] Nov 02 18:46:24.404814 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.404768 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:46:24.405311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.404857 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:46:24.460839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.460794 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-7bckb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:46:24.461006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.460869 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:46:24.515086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.515056 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-8flxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 02 18:46:24.515221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.515108 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 02 18:46:24.872724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.872688 5647 generic.go:356] "Generic (PLEG): container finished" podID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerID="06534d0ce220bbaa02c52f8d17e4bda7998c1bf7ce0e3f092f3302fb4c708737" exitCode=0 Nov 02 18:46:24.872939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.872852 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" event={"ID":"342fc860-018d-4d7b-9838-4578ab8e1aa9","Type":"ContainerDied","Data":"06534d0ce220bbaa02c52f8d17e4bda7998c1bf7ce0e3f092f3302fb4c708737"} Nov 02 18:46:24.872939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.872906 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" event={"ID":"342fc860-018d-4d7b-9838-4578ab8e1aa9","Type":"ContainerDied","Data":"63f63d257e9a7b32d67d0e9dc97528e66591f7638d79fa757ad6c0e99213488f"} Nov 02 18:46:24.872939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.872930 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63f63d257e9a7b32d67d0e9dc97528e66591f7638d79fa757ad6c0e99213488f" Nov 02 18:46:24.874374 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.874347 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:46:24.874708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.874688 5647 generic.go:356] "Generic (PLEG): container finished" podID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerID="0b76941a7c0fd94c23156d8c9018b479a92fc2d84cc6825ccebf214458578e11" exitCode=0 Nov 02 18:46:24.874956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.874810 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" event={"ID":"9decc592-ea53-42b3-9f3f-eeaba805f7a2","Type":"ContainerDied","Data":"0b76941a7c0fd94c23156d8c9018b479a92fc2d84cc6825ccebf214458578e11"} Nov 02 18:46:24.875057 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.874989 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" event={"ID":"9decc592-ea53-42b3-9f3f-eeaba805f7a2","Type":"ContainerDied","Data":"a85b8d71a99aa466ac1a2c96e85ce3969372b2ad2f19447b9a148035fc7debb9"} Nov 02 18:46:24.875057 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.875016 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a85b8d71a99aa466ac1a2c96e85ce3969372b2ad2f19447b9a148035fc7debb9" Nov 02 18:46:24.878907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.878880 5647 generic.go:356] "Generic (PLEG): container finished" podID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerID="ca296b09f6a8eaf937438a68356d52dedffb8e98a6c189dd9c4110b0956e6e4a" exitCode=0 Nov 02 18:46:24.879045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.878912 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:46:24.879045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.878967 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" event={"ID":"ca373f8d-d622-429a-944e-5508eae9e3fb","Type":"ContainerDied","Data":"ca296b09f6a8eaf937438a68356d52dedffb8e98a6c189dd9c4110b0956e6e4a"} Nov 02 18:46:24.895104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.895079 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:46:24.961971 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.961886 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.961971 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.961920 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.962222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.961998 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.962222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962034 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.962222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962205 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.962387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962245 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.962387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962261 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.962387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962271 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.962387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962328 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.962387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962360 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.962387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962371 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962399 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962432 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962460 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962488 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962535 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962569 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962570 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962594 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962656 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962679 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.962723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962712 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962803 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962849 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962897 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962981 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963014 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.962846 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963023 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963066 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data\") pod \"342fc860-018d-4d7b-9838-4578ab8e1aa9\" (UID: \"342fc860-018d-4d7b-9838-4578ab8e1aa9\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963107 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963145 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket\") pod \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\" (UID: \"9decc592-ea53-42b3-9f3f-eeaba805f7a2\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963183 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963244 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963281 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data\") pod \"ca373f8d-d622-429a-944e-5508eae9e3fb\" (UID: \"ca373f8d-d622-429a-944e-5508eae9e3fb\") " Nov 02 18:46:24.963388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963343 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:24.964827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963479 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.964827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963579 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.964827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963654 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.964827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963801 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.964827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.963892 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.964827 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.964165 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.965453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965425 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965648 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "kube-api-access-7lzmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965685 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965698 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965716 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca373f8d-d622-429a-944e-5508eae9e3fb-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965756 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965778 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.965709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965794 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965809 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965824 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965838 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965851 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965865 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/342fc860-018d-4d7b-9838-4578ab8e1aa9-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965878 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965892 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965905 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965921 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.965935 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:24.966968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.966517 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "kube-api-access-gf6dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:24.966968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.966920 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:46:24.967100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.966998 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy" pod "9decc592-ea53-42b3-9f3f-eeaba805f7a2" (UID: "9decc592-ea53-42b3-9f3f-eeaba805f7a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.967495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.967466 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:46:24.967581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.967521 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:24.967772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.967722 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:24.968186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.968169 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z" pod "ca373f8d-d622-429a-944e-5508eae9e3fb" (UID: "ca373f8d-d622-429a-944e-5508eae9e3fb"). InnerVolumeSpecName "kube-api-access-qft7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:24.968336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.968312 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:24.968607 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:24.968589 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy" pod "342fc860-018d-4d7b-9838-4578ab8e1aa9" (UID: "342fc860-018d-4d7b-9838-4578ab8e1aa9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:46:25.067679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067645 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067679 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067690 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf6dg\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-kube-api-access-gf6dg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067704 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067712 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qft7z\" (UniqueName: \"kubernetes.io/projected/ca373f8d-d622-429a-944e-5508eae9e3fb-kube-api-access-qft7z\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067720 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067749 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca373f8d-d622-429a-944e-5508eae9e3fb-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067765 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067780 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067791 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9decc592-ea53-42b3-9f3f-eeaba805f7a2-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067800 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzmz\" (UniqueName: \"kubernetes.io/projected/342fc860-018d-4d7b-9838-4578ab8e1aa9-kube-api-access-7lzmz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.067928 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.067808 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/342fc860-018d-4d7b-9838-4578ab8e1aa9-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:25.951695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.951644 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc" Nov 02 18:46:25.953411 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.953391 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" Nov 02 18:46:25.954143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.954120 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb" event={"ID":"ca373f8d-d622-429a-944e-5508eae9e3fb","Type":"ContainerDied","Data":"2758bdba4cedc5ec681d6b5c81d9049593f0987200c8ff40daa114b1ca4f0ae7"} Nov 02 18:46:25.954300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.954285 5647 scope.go:122] "RemoveContainer" containerID="ca296b09f6a8eaf937438a68356d52dedffb8e98a6c189dd9c4110b0956e6e4a" Nov 02 18:46:25.954563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:25.954548 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb" Nov 02 18:46:26.252888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.252784 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc"] Nov 02 18:46:26.258819 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.258787 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-8flxc"] Nov 02 18:46:26.384869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.384835 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb"] Nov 02 18:46:26.419517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.419469 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-7bckb"] Nov 02 18:46:26.498777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.498310 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb"] Nov 02 18:46:26.501336 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.501309 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-xqvvb"] Nov 02 18:46:26.809432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:26.807546 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerName="payload-processing" containerID="cri-o://17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5" gracePeriod=25 Nov 02 18:46:27.083004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:27.082925 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" path="/var/lib/kubelet/pods/342fc860-018d-4d7b-9838-4578ab8e1aa9/volumes" Nov 02 18:46:27.083412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:27.083399 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" path="/var/lib/kubelet/pods/9decc592-ea53-42b3-9f3f-eeaba805f7a2/volumes" Nov 02 18:46:27.083837 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:27.083823 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" path="/var/lib/kubelet/pods/ca373f8d-d622-429a-944e-5508eae9e3fb/volumes" Nov 02 18:46:31.434380 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.434343 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7"] Nov 02 18:46:31.435004 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.434988 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" containerName="payload-processing" Nov 02 18:46:31.435055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435007 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" containerName="payload-processing" Nov 02 18:46:31.435055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435045 5647 cpu_manager.go:414] "containerMap: removing container" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" Nov 02 18:46:31.435055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435051 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435060 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435066 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435081 5647 cpu_manager.go:414] "containerMap: removing container" podUID="32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" containerName="payload-pre-processing" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435087 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" containerName="payload-pre-processing" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435103 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435108 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435121 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d4a26bf4-531b-4a9f-a586-bb0e1f39a756" containerName="revoke-keys" Nov 02 18:46:31.435160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435126 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4a26bf4-531b-4a9f-a586-bb0e1f39a756" containerName="revoke-keys" Nov 02 18:46:31.435419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435229 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d6e7cc-d508-4f3b-91ed-c7cafdcba7b6" containerName="payload-pre-processing" Nov 02 18:46:31.435419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435237 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9decc592-ea53-42b3-9f3f-eeaba805f7a2" containerName="istio-proxy" Nov 02 18:46:31.435419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435249 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4a26bf4-531b-4a9f-a586-bb0e1f39a756" containerName="revoke-keys" Nov 02 18:46:31.435419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435261 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dbc72d7-bb45-4a9a-a69b-39c7e8503aaf" containerName="payload-processing" Nov 02 18:46:31.435419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435274 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="342fc860-018d-4d7b-9838-4578ab8e1aa9" containerName="istio-proxy" Nov 02 18:46:31.435419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.435281 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca373f8d-d622-429a-944e-5508eae9e3fb" containerName="istio-proxy" Nov 02 18:46:31.440305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.440285 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.499987 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.499952 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7"] Nov 02 18:46:31.543123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.543094 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:46:31.543305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.543157 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:46:31.586546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.586510 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsb67\" (UniqueName: \"kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.586776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.586555 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.586776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.586756 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.688214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.688125 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.688379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.688255 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsb67\" (UniqueName: \"kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.688379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.688283 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.688767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.688713 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.690858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.690839 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.746117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.746091 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsb67\" (UniqueName: \"kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67\") pod \"maas-api-revoke-sub-c1cb6108-tvqs7\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:31.753961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:31.753938 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:32.130906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:32.130875 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7"] Nov 02 18:46:32.132213 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:46:32.132182 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd86b039_4e46_411e_a0bb_d2edc87cd252.slice/crio-77c546a5f1134aa1cac2cdd717fd34560fba0ad44aa1bae54e5b9fda3d2fae15 WatchSource:0}: Error finding container 77c546a5f1134aa1cac2cdd717fd34560fba0ad44aa1bae54e5b9fda3d2fae15: Status 404 returned error can't find the container with id 77c546a5f1134aa1cac2cdd717fd34560fba0ad44aa1bae54e5b9fda3d2fae15 Nov 02 18:46:33.000877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:33.000838 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" event={"ID":"dd86b039-4e46-411e-a0bb-d2edc87cd252","Type":"ContainerStarted","Data":"7756afa58e6f0deab7b2f8d8f8ef87b962e3edce7b85ba02e64b05f93b2ae215"} Nov 02 18:46:33.001340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:33.000890 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" event={"ID":"dd86b039-4e46-411e-a0bb-d2edc87cd252","Type":"ContainerStarted","Data":"77c546a5f1134aa1cac2cdd717fd34560fba0ad44aa1bae54e5b9fda3d2fae15"} Nov 02 18:46:33.117793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:33.117717 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" podStartSLOduration=2.117700215 podStartE2EDuration="2.117700215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:33.116637536 +0000 UTC m=+4980.190164740" watchObservedRunningTime="2026-11-02 18:46:33.117700215 +0000 UTC m=+4980.191227419" Nov 02 18:46:34.009170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:34.009127 5647 generic.go:356] "Generic (PLEG): container finished" podID="dd86b039-4e46-411e-a0bb-d2edc87cd252" containerID="7756afa58e6f0deab7b2f8d8f8ef87b962e3edce7b85ba02e64b05f93b2ae215" exitCode=0 Nov 02 18:46:34.009582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:34.009215 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" event={"ID":"dd86b039-4e46-411e-a0bb-d2edc87cd252","Type":"ContainerDied","Data":"7756afa58e6f0deab7b2f8d8f8ef87b962e3edce7b85ba02e64b05f93b2ae215"} Nov 02 18:46:35.168854 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.168824 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:35.233417 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.233386 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca\") pod \"dd86b039-4e46-411e-a0bb-d2edc87cd252\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " Nov 02 18:46:35.233625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.233577 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca\") pod \"dd86b039-4e46-411e-a0bb-d2edc87cd252\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " Nov 02 18:46:35.233698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.233672 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67\" (UniqueName: \"kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67\") pod \"dd86b039-4e46-411e-a0bb-d2edc87cd252\" (UID: \"dd86b039-4e46-411e-a0bb-d2edc87cd252\") " Nov 02 18:46:35.233975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.233952 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca" pod "dd86b039-4e46-411e-a0bb-d2edc87cd252" (UID: "dd86b039-4e46-411e-a0bb-d2edc87cd252"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:35.234142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.234129 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:35.235593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.235570 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca" pod "dd86b039-4e46-411e-a0bb-d2edc87cd252" (UID: "dd86b039-4e46-411e-a0bb-d2edc87cd252"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:46:35.235836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.235813 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67" pod "dd86b039-4e46-411e-a0bb-d2edc87cd252" (UID: "dd86b039-4e46-411e-a0bb-d2edc87cd252"). InnerVolumeSpecName "kube-api-access-rsb67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:35.335573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.335480 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsb67\" (UniqueName: \"kubernetes.io/projected/dd86b039-4e46-411e-a0bb-d2edc87cd252-kube-api-access-rsb67\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:35.335573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.335514 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd86b039-4e46-411e-a0bb-d2edc87cd252-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:35.903905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:35.903855 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:46:35.947806 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:46:35.947762 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:46:36.023777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:36.023715 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" Nov 02 18:46:36.023777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:36.023773 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7" event={"ID":"dd86b039-4e46-411e-a0bb-d2edc87cd252","Type":"ContainerDied","Data":"77c546a5f1134aa1cac2cdd717fd34560fba0ad44aa1bae54e5b9fda3d2fae15"} Nov 02 18:46:36.024052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:36.023823 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c546a5f1134aa1cac2cdd717fd34560fba0ad44aa1bae54e5b9fda3d2fae15" Nov 02 18:46:37.613961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:37.613927 5647 scope.go:122] "RemoveContainer" containerID="0b76941a7c0fd94c23156d8c9018b479a92fc2d84cc6825ccebf214458578e11" Nov 02 18:46:37.624379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:37.624356 5647 scope.go:122] "RemoveContainer" containerID="1ed266a3d101f96036f12df81bce5ab7bf165b6ec4ec7275be2da820532d1621" Nov 02 18:46:37.634161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:37.634138 5647 scope.go:122] "RemoveContainer" containerID="06534d0ce220bbaa02c52f8d17e4bda7998c1bf7ce0e3f092f3302fb4c708737" Nov 02 18:46:38.409240 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.409208 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb"] Nov 02 18:46:38.409972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.409951 5647 cpu_manager.go:414] "containerMap: removing container" podUID="dd86b039-4e46-411e-a0bb-d2edc87cd252" containerName="revoke-keys" Nov 02 18:46:38.409972 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.409972 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd86b039-4e46-411e-a0bb-d2edc87cd252" containerName="revoke-keys" Nov 02 18:46:38.410163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.410099 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd86b039-4e46-411e-a0bb-d2edc87cd252" containerName="revoke-keys" Nov 02 18:46:38.415263 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.415243 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.468764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.468716 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb"] Nov 02 18:46:38.474893 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.474861 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcp5\" (UniqueName: \"kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.475056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.474939 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.475209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.475177 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.516167 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.516136 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:46:38.516363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.516221 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:46:38.576918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.576877 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.577175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.577099 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.577248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.577192 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcp5\" (UniqueName: \"kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.577926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.577900 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.579832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.579813 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.634535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.634499 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcp5\" (UniqueName: \"kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5\") pod \"maas-api-revoke-sub-86c6d29f-b8mnb\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:38.729155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:38.729069 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:39.106842 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:46:39.106808 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf51689_3cc6_4f12_afcf_044f0ba3fdb2.slice/crio-4c8151219271af8e34235bb3d27ec46d80cc093e25edaf107ed48791234c3191 WatchSource:0}: Error finding container 4c8151219271af8e34235bb3d27ec46d80cc093e25edaf107ed48791234c3191: Status 404 returned error can't find the container with id 4c8151219271af8e34235bb3d27ec46d80cc093e25edaf107ed48791234c3191 Nov 02 18:46:39.108255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:39.108230 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb"] Nov 02 18:46:40.051769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:40.051696 5647 generic.go:356] "Generic (PLEG): container finished" podID="adf51689-3cc6-4f12-afcf-044f0ba3fdb2" containerID="6231f704c4917a866b2ae120f5d09e533a5f6f76a02bb63cd7077c1cd70ad9fa" exitCode=0 Nov 02 18:46:40.052163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:40.051793 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" event={"ID":"adf51689-3cc6-4f12-afcf-044f0ba3fdb2","Type":"ContainerDied","Data":"6231f704c4917a866b2ae120f5d09e533a5f6f76a02bb63cd7077c1cd70ad9fa"} Nov 02 18:46:40.052163 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:40.051848 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" event={"ID":"adf51689-3cc6-4f12-afcf-044f0ba3fdb2","Type":"ContainerStarted","Data":"4c8151219271af8e34235bb3d27ec46d80cc093e25edaf107ed48791234c3191"} Nov 02 18:46:41.219920 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.219884 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:41.313133 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.313042 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca\") pod \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " Nov 02 18:46:41.313314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.313168 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5\" (UniqueName: \"kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5\") pod \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " Nov 02 18:46:41.313314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.313225 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca\") pod \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\" (UID: \"adf51689-3cc6-4f12-afcf-044f0ba3fdb2\") " Nov 02 18:46:41.313464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.313395 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca" pod "adf51689-3cc6-4f12-afcf-044f0ba3fdb2" (UID: "adf51689-3cc6-4f12-afcf-044f0ba3fdb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:41.313601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.313583 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:41.315495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.315466 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca" pod "adf51689-3cc6-4f12-afcf-044f0ba3fdb2" (UID: "adf51689-3cc6-4f12-afcf-044f0ba3fdb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:46:41.315605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.315497 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5" pod "adf51689-3cc6-4f12-afcf-044f0ba3fdb2" (UID: "adf51689-3cc6-4f12-afcf-044f0ba3fdb2"). InnerVolumeSpecName "kube-api-access-rfcp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:41.415343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.415303 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfcp5\" (UniqueName: \"kubernetes.io/projected/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-kube-api-access-rfcp5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:41.415343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:41.415337 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adf51689-3cc6-4f12-afcf-044f0ba3fdb2-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:42.066590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.066553 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" Nov 02 18:46:42.066820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.066591 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb" event={"ID":"adf51689-3cc6-4f12-afcf-044f0ba3fdb2","Type":"ContainerDied","Data":"4c8151219271af8e34235bb3d27ec46d80cc093e25edaf107ed48791234c3191"} Nov 02 18:46:42.066820 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.066640 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8151219271af8e34235bb3d27ec46d80cc093e25edaf107ed48791234c3191" Nov 02 18:46:42.198631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.198599 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6"] Nov 02 18:46:42.199461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.199435 5647 cpu_manager.go:414] "containerMap: removing container" podUID="adf51689-3cc6-4f12-afcf-044f0ba3fdb2" containerName="revoke-keys" Nov 02 18:46:42.199461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.199460 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf51689-3cc6-4f12-afcf-044f0ba3fdb2" containerName="revoke-keys" Nov 02 18:46:42.199687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.199561 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf51689-3cc6-4f12-afcf-044f0ba3fdb2" containerName="revoke-keys" Nov 02 18:46:42.203236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.203212 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.257367 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.257334 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:46:42.257857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.257634 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:46:42.258764 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.258711 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6"] Nov 02 18:46:42.326093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.325994 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.326093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.326045 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.326304 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.326198 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n77l7\" (UniqueName: \"kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.427364 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.427325 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.427584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.427377 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.427584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.427421 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n77l7\" (UniqueName: \"kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.428185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.428154 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.429901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.429882 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.485187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.485147 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n77l7\" (UniqueName: \"kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7\") pod \"maas-api-revoke-sub-88e19cbb-vjjm6\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.518181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.518150 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:42.898938 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:46:42.898890 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e175876_c12c_433e_ad08_432325ac918c.slice/crio-773cdf20160370e30d86a75854905249db841605febcf9d0de500417d254e232 WatchSource:0}: Error finding container 773cdf20160370e30d86a75854905249db841605febcf9d0de500417d254e232: Status 404 returned error can't find the container with id 773cdf20160370e30d86a75854905249db841605febcf9d0de500417d254e232 Nov 02 18:46:42.899407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:42.899383 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6"] Nov 02 18:46:43.082098 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:43.082061 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" event={"ID":"6e175876-c12c-433e-ad08-432325ac918c","Type":"ContainerStarted","Data":"773cdf20160370e30d86a75854905249db841605febcf9d0de500417d254e232"} Nov 02 18:46:44.082265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:44.082165 5647 generic.go:356] "Generic (PLEG): container finished" podID="6e175876-c12c-433e-ad08-432325ac918c" containerID="90df631a2581ef8858b5f2063e45c71c0f85e8f83d1e5fc0d4152c63237758f9" exitCode=0 Nov 02 18:46:44.082265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:44.082253 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" event={"ID":"6e175876-c12c-433e-ad08-432325ac918c","Type":"ContainerDied","Data":"90df631a2581ef8858b5f2063e45c71c0f85e8f83d1e5fc0d4152c63237758f9"} Nov 02 18:46:45.249573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.249544 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:45.366371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.366286 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca\") pod \"6e175876-c12c-433e-ad08-432325ac918c\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " Nov 02 18:46:45.366371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.366350 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7\" (UniqueName: \"kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7\") pod \"6e175876-c12c-433e-ad08-432325ac918c\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " Nov 02 18:46:45.366625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.366531 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca\") pod \"6e175876-c12c-433e-ad08-432325ac918c\" (UID: \"6e175876-c12c-433e-ad08-432325ac918c\") " Nov 02 18:46:45.367657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.367624 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca" pod "6e175876-c12c-433e-ad08-432325ac918c" (UID: "6e175876-c12c-433e-ad08-432325ac918c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:45.370415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.370370 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca" pod "6e175876-c12c-433e-ad08-432325ac918c" (UID: "6e175876-c12c-433e-ad08-432325ac918c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:46:45.370638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.370608 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7" pod "6e175876-c12c-433e-ad08-432325ac918c" (UID: "6e175876-c12c-433e-ad08-432325ac918c"). InnerVolumeSpecName "kube-api-access-n77l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:45.468570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.468522 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e175876-c12c-433e-ad08-432325ac918c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:45.468570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.468567 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e175876-c12c-433e-ad08-432325ac918c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:45.468845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.468595 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n77l7\" (UniqueName: \"kubernetes.io/projected/6e175876-c12c-433e-ad08-432325ac918c-kube-api-access-n77l7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:45.904373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:45.904331 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:46:46.096975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:46.096943 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" Nov 02 18:46:46.096975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:46.096960 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6" event={"ID":"6e175876-c12c-433e-ad08-432325ac918c","Type":"ContainerDied","Data":"773cdf20160370e30d86a75854905249db841605febcf9d0de500417d254e232"} Nov 02 18:46:46.097222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:46.097004 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="773cdf20160370e30d86a75854905249db841605febcf9d0de500417d254e232" Nov 02 18:46:46.967585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:46.967557 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:46:46.995257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:46.995234 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:46:47.104850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.104818 5647 generic.go:356] "Generic (PLEG): container finished" podID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerID="17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5" exitCode=0 Nov 02 18:46:47.105059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.104861 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" event={"ID":"784b5e8a-a63d-4d21-8a57-8d2f29d095f3","Type":"ContainerDied","Data":"17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5"} Nov 02 18:46:47.105059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.104936 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" event={"ID":"784b5e8a-a63d-4d21-8a57-8d2f29d095f3","Type":"ContainerDied","Data":"9c508fc86b0b8a204f85905ab12e1c3a5f2ca11d952e91b7d6bb6b3c552a4123"} Nov 02 18:46:47.105059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.104869 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb" Nov 02 18:46:47.105059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.104968 5647 scope.go:122] "RemoveContainer" containerID="17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5" Nov 02 18:46:47.112857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.112811 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume\") pod \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " Nov 02 18:46:47.113032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.113009 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm\" (UniqueName: \"kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm\") pod \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\" (UID: \"784b5e8a-a63d-4d21-8a57-8d2f29d095f3\") " Nov 02 18:46:47.113391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.113365 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume" pod "784b5e8a-a63d-4d21-8a57-8d2f29d095f3" (UID: "784b5e8a-a63d-4d21-8a57-8d2f29d095f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:47.113891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.113865 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:47.115936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.115831 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm" pod "784b5e8a-a63d-4d21-8a57-8d2f29d095f3" (UID: "784b5e8a-a63d-4d21-8a57-8d2f29d095f3"). InnerVolumeSpecName "kube-api-access-2srhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:47.118176 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.118154 5647 scope.go:122] "RemoveContainer" containerID="17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5" Nov 02 18:46:47.118570 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:46:47.118548 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5\": container with ID starting with 17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5 not found: ID does not exist" containerID="17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5" Nov 02 18:46:47.118664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.118577 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5"} err="failed to get container status \"17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5\": rpc error: code = NotFound desc = could not find container \"17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5\": container with ID starting with 17921989bc12d76b58bf402169a96eda03054a9ecce2344a7cc94a018bcba3c5 not found: ID does not exist" Nov 02 18:46:47.215715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.215678 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2srhm\" (UniqueName: \"kubernetes.io/projected/784b5e8a-a63d-4d21-8a57-8d2f29d095f3-kube-api-access-2srhm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:47.587112 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.581594 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb"] Nov 02 18:46:47.599766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:47.599371 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-134555-7b5f4dc6c9-h7rkb"] Nov 02 18:46:49.081940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.081904 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" path="/var/lib/kubelet/pods/784b5e8a-a63d-4d21-8a57-8d2f29d095f3/volumes" Nov 02 18:46:49.134869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.134834 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8"] Nov 02 18:46:49.135401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.135388 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6e175876-c12c-433e-ad08-432325ac918c" containerName="revoke-keys" Nov 02 18:46:49.135449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.135403 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e175876-c12c-433e-ad08-432325ac918c" containerName="revoke-keys" Nov 02 18:46:49.135449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.135438 5647 cpu_manager.go:414] "containerMap: removing container" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerName="payload-processing" Nov 02 18:46:49.135449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.135444 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerName="payload-processing" Nov 02 18:46:49.135550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.135536 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="784b5e8a-a63d-4d21-8a57-8d2f29d095f3" containerName="payload-processing" Nov 02 18:46:49.135584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.135550 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e175876-c12c-433e-ad08-432325ac918c" containerName="revoke-keys" Nov 02 18:46:49.140533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.140513 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.162817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.162790 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjhb\" (UniqueName: \"kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.162959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.162913 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.163214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.163196 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.195332 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.195302 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8"] Nov 02 18:46:49.242795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.242728 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:46:49.242979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.242807 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:46:49.263933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.263886 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.264153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.263996 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.264153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.264053 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjhb\" (UniqueName: \"kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.264659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.264634 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.266548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.266529 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.321901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.321867 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjhb\" (UniqueName: \"kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb\") pod \"maas-api-revoke-sub-d495f970-g8xg8\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.454451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.454414 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:49.830699 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:49.830666 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8"] Nov 02 18:46:49.832005 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:46:49.831981 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68586401_3cfb_4d7e_9a0b_22f2e45a100e.slice/crio-f7620f7adfbf95029a1d79ff750ddecae85b4832fad0e686824b284599cc0232 WatchSource:0}: Error finding container f7620f7adfbf95029a1d79ff750ddecae85b4832fad0e686824b284599cc0232: Status 404 returned error can't find the container with id f7620f7adfbf95029a1d79ff750ddecae85b4832fad0e686824b284599cc0232 Nov 02 18:46:50.126350 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:50.126317 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" event={"ID":"68586401-3cfb-4d7e-9a0b-22f2e45a100e","Type":"ContainerStarted","Data":"1481554347aa3603216b412e77e035d4086f6c1606fa55a8ad03b60d54603d62"} Nov 02 18:46:50.126676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:50.126375 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" event={"ID":"68586401-3cfb-4d7e-9a0b-22f2e45a100e","Type":"ContainerStarted","Data":"f7620f7adfbf95029a1d79ff750ddecae85b4832fad0e686824b284599cc0232"} Nov 02 18:46:50.242270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:50.242218 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" podStartSLOduration=1.242203204 podStartE2EDuration="1.242203204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:50.240177381 +0000 UTC m=+4997.313704584" watchObservedRunningTime="2026-11-02 18:46:50.242203204 +0000 UTC m=+4997.315730448" Nov 02 18:46:51.133949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:51.133910 5647 generic.go:356] "Generic (PLEG): container finished" podID="68586401-3cfb-4d7e-9a0b-22f2e45a100e" containerID="1481554347aa3603216b412e77e035d4086f6c1606fa55a8ad03b60d54603d62" exitCode=0 Nov 02 18:46:51.134414 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:51.134000 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" event={"ID":"68586401-3cfb-4d7e-9a0b-22f2e45a100e","Type":"ContainerDied","Data":"1481554347aa3603216b412e77e035d4086f6c1606fa55a8ad03b60d54603d62"} Nov 02 18:46:51.999119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:51.999079 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht"] Nov 02 18:46:52.003212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.003192 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.061813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.061770 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht"] Nov 02 18:46:52.098340 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.098294 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mxpw\" (UniqueName: \"kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.098550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.098442 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.098550 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.098503 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.200334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.200300 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.200803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.200468 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mxpw\" (UniqueName: \"kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.200803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.200523 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.200990 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.200965 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.202983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.202955 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.259483 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.259398 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mxpw\" (UniqueName: \"kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw\") pod \"maas-api-revoke-sub-b3b13b50-jjpht\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.309405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.309380 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:52.316728 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.316702 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:52.403450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.402713 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb\" (UniqueName: \"kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb\") pod \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " Nov 02 18:46:52.403450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.402924 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca\") pod \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " Nov 02 18:46:52.403450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.403006 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca\") pod \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\" (UID: \"68586401-3cfb-4d7e-9a0b-22f2e45a100e\") " Nov 02 18:46:52.403450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.403393 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca" pod "68586401-3cfb-4d7e-9a0b-22f2e45a100e" (UID: "68586401-3cfb-4d7e-9a0b-22f2e45a100e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:52.406200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.406165 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca" pod "68586401-3cfb-4d7e-9a0b-22f2e45a100e" (UID: "68586401-3cfb-4d7e-9a0b-22f2e45a100e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:46:52.406378 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.406358 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb" pod "68586401-3cfb-4d7e-9a0b-22f2e45a100e" (UID: "68586401-3cfb-4d7e-9a0b-22f2e45a100e"). InnerVolumeSpecName "kube-api-access-kdjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:52.505260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.505229 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:52.505260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.505259 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68586401-3cfb-4d7e-9a0b-22f2e45a100e-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:52.505488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.505269 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjhb\" (UniqueName: \"kubernetes.io/projected/68586401-3cfb-4d7e-9a0b-22f2e45a100e-kube-api-access-kdjhb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:52.693460 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:52.693423 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht"] Nov 02 18:46:52.694794 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:46:52.694762 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e48443_53cc_49fd_bfa0_8b8fe22683ec.slice/crio-76f25a15da48a6dd8b93c8fe5bf2dccbfa77129a4da8c66ebf5983fbab771142 WatchSource:0}: Error finding container 76f25a15da48a6dd8b93c8fe5bf2dccbfa77129a4da8c66ebf5983fbab771142: Status 404 returned error can't find the container with id 76f25a15da48a6dd8b93c8fe5bf2dccbfa77129a4da8c66ebf5983fbab771142 Nov 02 18:46:53.147803 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:53.147760 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" event={"ID":"68586401-3cfb-4d7e-9a0b-22f2e45a100e","Type":"ContainerDied","Data":"f7620f7adfbf95029a1d79ff750ddecae85b4832fad0e686824b284599cc0232"} Nov 02 18:46:53.148055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:53.147828 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7620f7adfbf95029a1d79ff750ddecae85b4832fad0e686824b284599cc0232" Nov 02 18:46:53.148055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:53.147774 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:46:53.149639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:53.149606 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" event={"ID":"98e48443-53cc-49fd-bfa0-8b8fe22683ec","Type":"ContainerStarted","Data":"f3b65b5a870622c1bd12bff8c1e1b9fe172372d574bb14c790b01b865d7593d9"} Nov 02 18:46:53.149821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:53.149660 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" event={"ID":"98e48443-53cc-49fd-bfa0-8b8fe22683ec","Type":"ContainerStarted","Data":"76f25a15da48a6dd8b93c8fe5bf2dccbfa77129a4da8c66ebf5983fbab771142"} Nov 02 18:46:53.266783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:53.266666 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" podStartSLOduration=2.266652281 podStartE2EDuration="2.266652281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:46:53.264927045 +0000 UTC m=+5000.338454249" watchObservedRunningTime="2026-11-02 18:46:53.266652281 +0000 UTC m=+5000.340179462" Nov 02 18:46:54.157306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:54.157272 5647 generic.go:356] "Generic (PLEG): container finished" podID="98e48443-53cc-49fd-bfa0-8b8fe22683ec" containerID="f3b65b5a870622c1bd12bff8c1e1b9fe172372d574bb14c790b01b865d7593d9" exitCode=0 Nov 02 18:46:54.157497 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:54.157365 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" event={"ID":"98e48443-53cc-49fd-bfa0-8b8fe22683ec","Type":"ContainerDied","Data":"f3b65b5a870622c1bd12bff8c1e1b9fe172372d574bb14c790b01b865d7593d9"} Nov 02 18:46:55.312145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.312120 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:55.338516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.338475 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca\") pod \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " Nov 02 18:46:55.338706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.338651 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca\") pod \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " Nov 02 18:46:55.338821 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.338794 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw\" (UniqueName: \"kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw\") pod \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\" (UID: \"98e48443-53cc-49fd-bfa0-8b8fe22683ec\") " Nov 02 18:46:55.338901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.338873 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca" pod "98e48443-53cc-49fd-bfa0-8b8fe22683ec" (UID: "98e48443-53cc-49fd-bfa0-8b8fe22683ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:46:55.339253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.339234 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:55.341233 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.341203 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw" pod "98e48443-53cc-49fd-bfa0-8b8fe22683ec" (UID: "98e48443-53cc-49fd-bfa0-8b8fe22683ec"). InnerVolumeSpecName "kube-api-access-5mxpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:46:55.341409 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.341378 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca" pod "98e48443-53cc-49fd-bfa0-8b8fe22683ec" (UID: "98e48443-53cc-49fd-bfa0-8b8fe22683ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:46:55.440664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.440588 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e48443-53cc-49fd-bfa0-8b8fe22683ec-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:55.440664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:55.440616 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mxpw\" (UniqueName: \"kubernetes.io/projected/98e48443-53cc-49fd-bfa0-8b8fe22683ec-kube-api-access-5mxpw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:46:56.171220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:56.171186 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" Nov 02 18:46:56.171220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:56.171209 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht" event={"ID":"98e48443-53cc-49fd-bfa0-8b8fe22683ec","Type":"ContainerDied","Data":"76f25a15da48a6dd8b93c8fe5bf2dccbfa77129a4da8c66ebf5983fbab771142"} Nov 02 18:46:56.171456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:46:56.171256 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76f25a15da48a6dd8b93c8fe5bf2dccbfa77129a4da8c66ebf5983fbab771142" Nov 02 18:47:00.714853 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.714814 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w"] Nov 02 18:47:00.715419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.715405 5647 cpu_manager.go:414] "containerMap: removing container" podUID="68586401-3cfb-4d7e-9a0b-22f2e45a100e" containerName="revoke-keys" Nov 02 18:47:00.715470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.715420 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68586401-3cfb-4d7e-9a0b-22f2e45a100e" containerName="revoke-keys" Nov 02 18:47:00.715470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.715451 5647 cpu_manager.go:414] "containerMap: removing container" podUID="98e48443-53cc-49fd-bfa0-8b8fe22683ec" containerName="revoke-keys" Nov 02 18:47:00.715470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.715456 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e48443-53cc-49fd-bfa0-8b8fe22683ec" containerName="revoke-keys" Nov 02 18:47:00.715572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.715538 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e48443-53cc-49fd-bfa0-8b8fe22683ec" containerName="revoke-keys" Nov 02 18:47:00.715572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.715553 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="68586401-3cfb-4d7e-9a0b-22f2e45a100e" containerName="revoke-keys" Nov 02 18:47:00.719174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.719158 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.773798 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.773765 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w"] Nov 02 18:47:00.809706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.809671 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.809952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.809838 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.809952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.809886 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55nc\" (UniqueName: \"kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.821026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.820990 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:47:00.821026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.821023 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:47:00.911226 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.911190 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55nc\" (UniqueName: \"kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.911413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.911320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.911413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.911380 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.912013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.911989 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.913983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.913961 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:00.969508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:00.969439 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55nc\" (UniqueName: \"kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc\") pod \"maas-api-revoke-sub-8ee8b00e-2vm7w\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:01.032799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:01.032765 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:01.410329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:01.410300 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w"] Nov 02 18:47:01.411562 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:47:01.411534 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29c8c83_a731_45f4_8a15_d5a4f758091a.slice/crio-208275adb9a2cddbbe984bcedee1248e5b4947f7760a0bc1c18401dca46d0598 WatchSource:0}: Error finding container 208275adb9a2cddbbe984bcedee1248e5b4947f7760a0bc1c18401dca46d0598: Status 404 returned error can't find the container with id 208275adb9a2cddbbe984bcedee1248e5b4947f7760a0bc1c18401dca46d0598 Nov 02 18:47:02.213252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.213208 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" event={"ID":"b29c8c83-a731-45f4-8a15-d5a4f758091a","Type":"ContainerStarted","Data":"3dd07989c1ffa78ff2e840372a67a64b334c2c3b9953b27608ff2cd9ba711820"} Nov 02 18:47:02.213252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.213282 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" event={"ID":"b29c8c83-a731-45f4-8a15-d5a4f758091a","Type":"ContainerStarted","Data":"208275adb9a2cddbbe984bcedee1248e5b4947f7760a0bc1c18401dca46d0598"} Nov 02 18:47:02.334205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.334133 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" podStartSLOduration=2.334116806 podStartE2EDuration="2.334116806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:02.33010872 +0000 UTC m=+5009.403635924" watchObservedRunningTime="2026-11-02 18:47:02.334116806 +0000 UTC m=+5009.407644009" Nov 02 18:47:02.678205 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.678166 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm"] Nov 02 18:47:02.682620 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.682596 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.740880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.740845 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm"] Nov 02 18:47:02.839307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.839274 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.839501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.839324 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.839556 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.839512 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztcw\" (UniqueName: \"kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.941212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.941132 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztcw\" (UniqueName: \"kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.941373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.941241 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.941373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.941266 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.941960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.941942 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.944012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.943977 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:02.998906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:02.998871 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztcw\" (UniqueName: \"kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw\") pod \"maas-api-revoke-sub-0ea8f49a-5xwvm\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:03.220680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.220592 5647 generic.go:356] "Generic (PLEG): container finished" podID="b29c8c83-a731-45f4-8a15-d5a4f758091a" containerID="3dd07989c1ffa78ff2e840372a67a64b334c2c3b9953b27608ff2cd9ba711820" exitCode=0 Nov 02 18:47:03.220680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.220670 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" event={"ID":"b29c8c83-a731-45f4-8a15-d5a4f758091a","Type":"ContainerDied","Data":"3dd07989c1ffa78ff2e840372a67a64b334c2c3b9953b27608ff2cd9ba711820"} Nov 02 18:47:03.297429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.297398 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:03.675671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.675640 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm"] Nov 02 18:47:03.676721 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:47:03.676689 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2b5cd6_c2d7_48bb_9956_e264c8679d80.slice/crio-ab2bc0a8f19279c9561f3f995e3aff7d6c33069c094fc9ff90c46c65715cf1c1 WatchSource:0}: Error finding container ab2bc0a8f19279c9561f3f995e3aff7d6c33069c094fc9ff90c46c65715cf1c1: Status 404 returned error can't find the container with id ab2bc0a8f19279c9561f3f995e3aff7d6c33069c094fc9ff90c46c65715cf1c1 Nov 02 18:47:03.684498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.684474 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc"] Nov 02 18:47:03.684687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.684667 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" podUID="1136cf8f-37c0-4b8a-b231-3cb9c699ae39" containerName="maas-api" containerID="cri-o://d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001" gracePeriod=30 Nov 02 18:47:03.997842 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:03.997817 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:47:04.089249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.088120 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\") pod \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " Nov 02 18:47:04.089249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.088219 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg\" (UniqueName: \"kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg\") pod \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " Nov 02 18:47:04.089249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.088289 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs\") pod \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " Nov 02 18:47:04.089249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.088501 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca\") pod \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\" (UID: \"1136cf8f-37c0-4b8a-b231-3cb9c699ae39\") " Nov 02 18:47:04.100209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.099245 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca" pod "1136cf8f-37c0-4b8a-b231-3cb9c699ae39" (UID: "1136cf8f-37c0-4b8a-b231-3cb9c699ae39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:04.116251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.116215 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls" pod "1136cf8f-37c0-4b8a-b231-3cb9c699ae39" (UID: "1136cf8f-37c0-4b8a-b231-3cb9c699ae39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:47:04.117018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.116984 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg" pod "1136cf8f-37c0-4b8a-b231-3cb9c699ae39" (UID: "1136cf8f-37c0-4b8a-b231-3cb9c699ae39"). InnerVolumeSpecName "kube-api-access-mrxbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:04.118665 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.118639 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs" pod "1136cf8f-37c0-4b8a-b231-3cb9c699ae39" (UID: "1136cf8f-37c0-4b8a-b231-3cb9c699ae39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:47:04.190642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.190542 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.190642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.190579 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.190642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.190593 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrxbg\" (UniqueName: \"kubernetes.io/projected/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-kube-api-access-mrxbg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.190642 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.190606 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1136cf8f-37c0-4b8a-b231-3cb9c699ae39-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.259160 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.259118 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" event={"ID":"df2b5cd6-c2d7-48bb-9956-e264c8679d80","Type":"ContainerStarted","Data":"3c3a63257536a5aef4ca6d7a4379a3b340e821ff978110d058792de8c4d164a3"} Nov 02 18:47:04.259691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.259187 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" event={"ID":"df2b5cd6-c2d7-48bb-9956-e264c8679d80","Type":"ContainerStarted","Data":"ab2bc0a8f19279c9561f3f995e3aff7d6c33069c094fc9ff90c46c65715cf1c1"} Nov 02 18:47:04.262406 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.262377 5647 generic.go:356] "Generic (PLEG): container finished" podID="1136cf8f-37c0-4b8a-b231-3cb9c699ae39" containerID="d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001" exitCode=0 Nov 02 18:47:04.262559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.262428 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" Nov 02 18:47:04.262559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.262453 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" event={"ID":"1136cf8f-37c0-4b8a-b231-3cb9c699ae39","Type":"ContainerDied","Data":"d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001"} Nov 02 18:47:04.262559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.262501 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc" event={"ID":"1136cf8f-37c0-4b8a-b231-3cb9c699ae39","Type":"ContainerDied","Data":"9384c71efcab6f2aadf92c5ac8fba63533fa962f0fa523d481da777858964cd1"} Nov 02 18:47:04.262559 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.262525 5647 scope.go:122] "RemoveContainer" containerID="d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001" Nov 02 18:47:04.276224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.276199 5647 scope.go:122] "RemoveContainer" containerID="d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001" Nov 02 18:47:04.276642 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:04.276613 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001\": container with ID starting with d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001 not found: ID does not exist" containerID="d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001" Nov 02 18:47:04.276787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.276649 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001"} err="failed to get container status \"d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001\": rpc error: code = NotFound desc = could not find container \"d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001\": container with ID starting with d315188dd1ce21a88ec43b8cc0e1130bf35ddc131738bbec3191e85aadf65001 not found: ID does not exist" Nov 02 18:47:04.379060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.378997 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" podStartSLOduration=2.37897669 podStartE2EDuration="2.37897669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:04.374312916 +0000 UTC m=+5011.447840138" watchObservedRunningTime="2026-11-02 18:47:04.37897669 +0000 UTC m=+5011.452503893" Nov 02 18:47:04.400178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.400150 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:04.487795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.487646 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc"] Nov 02 18:47:04.491483 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.491459 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8f40154e-5f7d87dfbb-kzzsc"] Nov 02 18:47:04.493289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.493270 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc\" (UniqueName: \"kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc\") pod \"b29c8c83-a731-45f4-8a15-d5a4f758091a\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " Nov 02 18:47:04.493381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.493362 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca\") pod \"b29c8c83-a731-45f4-8a15-d5a4f758091a\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " Nov 02 18:47:04.493544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.493528 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca\") pod \"b29c8c83-a731-45f4-8a15-d5a4f758091a\" (UID: \"b29c8c83-a731-45f4-8a15-d5a4f758091a\") " Nov 02 18:47:04.493904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.493875 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca" pod "b29c8c83-a731-45f4-8a15-d5a4f758091a" (UID: "b29c8c83-a731-45f4-8a15-d5a4f758091a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:04.494088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.494072 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.495494 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.495470 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc" pod "b29c8c83-a731-45f4-8a15-d5a4f758091a" (UID: "b29c8c83-a731-45f4-8a15-d5a4f758091a"). InnerVolumeSpecName "kube-api-access-f55nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:04.495839 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.495816 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca" pod "b29c8c83-a731-45f4-8a15-d5a4f758091a" (UID: "b29c8c83-a731-45f4-8a15-d5a4f758091a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:47:04.595224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.595191 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f55nc\" (UniqueName: \"kubernetes.io/projected/b29c8c83-a731-45f4-8a15-d5a4f758091a-kube-api-access-f55nc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.595224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.595224 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b29c8c83-a731-45f4-8a15-d5a4f758091a-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:04.768716 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.768633 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl"] Nov 02 18:47:04.769250 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.769234 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b29c8c83-a731-45f4-8a15-d5a4f758091a" containerName="revoke-keys" Nov 02 18:47:04.769294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.769253 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29c8c83-a731-45f4-8a15-d5a4f758091a" containerName="revoke-keys" Nov 02 18:47:04.769294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.769271 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1136cf8f-37c0-4b8a-b231-3cb9c699ae39" containerName="maas-api" Nov 02 18:47:04.769294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.769276 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1136cf8f-37c0-4b8a-b231-3cb9c699ae39" containerName="maas-api" Nov 02 18:47:04.769392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.769377 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29c8c83-a731-45f4-8a15-d5a4f758091a" containerName="revoke-keys" Nov 02 18:47:04.769392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.769385 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1136cf8f-37c0-4b8a-b231-3cb9c699ae39" containerName="maas-api" Nov 02 18:47:04.772777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.772762 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:04.829136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.829098 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl"] Nov 02 18:47:04.910783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.910726 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:04.910783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.910784 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfk28\" (UniqueName: \"kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:04.911020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:04.910939 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.012260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.012205 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.012260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.012255 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfk28\" (UniqueName: \"kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.012544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.012353 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.013113 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.013085 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.014845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.014828 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.070524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.070452 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfk28\" (UniqueName: \"kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28\") pod \"maas-api-revoke-sub-571e3943-s78fl\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.082581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.082530 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1136cf8f-37c0-4b8a-b231-3cb9c699ae39" path="/var/lib/kubelet/pods/1136cf8f-37c0-4b8a-b231-3cb9c699ae39/volumes" Nov 02 18:47:05.086342 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.086319 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:05.269870 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.269830 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" event={"ID":"b29c8c83-a731-45f4-8a15-d5a4f758091a","Type":"ContainerDied","Data":"208275adb9a2cddbbe984bcedee1248e5b4947f7760a0bc1c18401dca46d0598"} Nov 02 18:47:05.270347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.269896 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208275adb9a2cddbbe984bcedee1248e5b4947f7760a0bc1c18401dca46d0598" Nov 02 18:47:05.270347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.269842 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w" Nov 02 18:47:05.271747 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.271702 5647 generic.go:356] "Generic (PLEG): container finished" podID="df2b5cd6-c2d7-48bb-9956-e264c8679d80" containerID="3c3a63257536a5aef4ca6d7a4379a3b340e821ff978110d058792de8c4d164a3" exitCode=0 Nov 02 18:47:05.271890 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.271765 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" event={"ID":"df2b5cd6-c2d7-48bb-9956-e264c8679d80","Type":"ContainerDied","Data":"3c3a63257536a5aef4ca6d7a4379a3b340e821ff978110d058792de8c4d164a3"} Nov 02 18:47:05.475093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:05.475056 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl"] Nov 02 18:47:05.476393 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:47:05.476368 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7391edd6_f08f_4e0b_a147_cdc2ac6d0bf4.slice/crio-85de5db8f11a33bbc0911b30d43ed3a760a3e9ac6f2c1fde621b1d2d05637ea5 WatchSource:0}: Error finding container 85de5db8f11a33bbc0911b30d43ed3a760a3e9ac6f2c1fde621b1d2d05637ea5: Status 404 returned error can't find the container with id 85de5db8f11a33bbc0911b30d43ed3a760a3e9ac6f2c1fde621b1d2d05637ea5 Nov 02 18:47:05.949435 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:05.949391 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:47:06.281515 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.281473 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" event={"ID":"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4","Type":"ContainerStarted","Data":"dceb129272a174bb230887b70246150cb0d34fd362280765a061077a950d1580"} Nov 02 18:47:06.282056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.281538 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" event={"ID":"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4","Type":"ContainerStarted","Data":"85de5db8f11a33bbc0911b30d43ed3a760a3e9ac6f2c1fde621b1d2d05637ea5"} Nov 02 18:47:06.399013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.398960 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" podStartSLOduration=2.398945059 podStartE2EDuration="2.398945059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:47:06.396620014 +0000 UTC m=+5013.470147218" watchObservedRunningTime="2026-11-02 18:47:06.398945059 +0000 UTC m=+5013.472472262" Nov 02 18:47:06.447284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.447258 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:06.632296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.632266 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca\") pod \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " Nov 02 18:47:06.632487 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.632339 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca\") pod \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " Nov 02 18:47:06.632487 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.632385 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw\" (UniqueName: \"kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw\") pod \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\" (UID: \"df2b5cd6-c2d7-48bb-9956-e264c8679d80\") " Nov 02 18:47:06.632699 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.632674 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca" pod "df2b5cd6-c2d7-48bb-9956-e264c8679d80" (UID: "df2b5cd6-c2d7-48bb-9956-e264c8679d80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:06.632971 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.632954 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:06.634464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.634441 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca" pod "df2b5cd6-c2d7-48bb-9956-e264c8679d80" (UID: "df2b5cd6-c2d7-48bb-9956-e264c8679d80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:47:06.634548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.634513 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw" pod "df2b5cd6-c2d7-48bb-9956-e264c8679d80" (UID: "df2b5cd6-c2d7-48bb-9956-e264c8679d80"). InnerVolumeSpecName "kube-api-access-4ztcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:06.734335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.734299 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2b5cd6-c2d7-48bb-9956-e264c8679d80-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:06.734335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:06.734328 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ztcw\" (UniqueName: \"kubernetes.io/projected/df2b5cd6-c2d7-48bb-9956-e264c8679d80-kube-api-access-4ztcw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:07.291436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:07.291397 5647 generic.go:356] "Generic (PLEG): container finished" podID="7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" containerID="dceb129272a174bb230887b70246150cb0d34fd362280765a061077a950d1580" exitCode=0 Nov 02 18:47:07.291962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:07.291473 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" event={"ID":"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4","Type":"ContainerDied","Data":"dceb129272a174bb230887b70246150cb0d34fd362280765a061077a950d1580"} Nov 02 18:47:07.293023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:07.292998 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" event={"ID":"df2b5cd6-c2d7-48bb-9956-e264c8679d80","Type":"ContainerDied","Data":"ab2bc0a8f19279c9561f3f995e3aff7d6c33069c094fc9ff90c46c65715cf1c1"} Nov 02 18:47:07.293023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:07.293023 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm" Nov 02 18:47:07.293237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:07.293038 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab2bc0a8f19279c9561f3f995e3aff7d6c33069c094fc9ff90c46c65715cf1c1" Nov 02 18:47:08.105330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.105298 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh"] Nov 02 18:47:08.105582 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.105552 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" containerID="cri-o://b88b2cbe1ed82a372772084d43b350b3814ffb77e77e1622d2e65e55d80a3151" gracePeriod=30 Nov 02 18:47:08.461802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.461776 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:08.558387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.558359 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28\" (UniqueName: \"kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28\") pod \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " Nov 02 18:47:08.558605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.558445 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca\") pod \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " Nov 02 18:47:08.558605 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.558566 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca\") pod \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\" (UID: \"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4\") " Nov 02 18:47:08.558971 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.558937 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca" pod "7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" (UID: "7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:08.561236 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.561209 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca" pod "7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" (UID: "7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:47:08.561353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.561269 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28" pod "7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" (UID: "7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4"). InnerVolumeSpecName "kube-api-access-mfk28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:08.666426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.666331 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfk28\" (UniqueName: \"kubernetes.io/projected/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-kube-api-access-mfk28\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:08.666426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.666376 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:08.666426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:08.666392 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:09.312297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:09.312265 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" Nov 02 18:47:09.312297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:09.312291 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl" event={"ID":"7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4","Type":"ContainerDied","Data":"85de5db8f11a33bbc0911b30d43ed3a760a3e9ac6f2c1fde621b1d2d05637ea5"} Nov 02 18:47:09.312575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:09.312338 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85de5db8f11a33bbc0911b30d43ed3a760a3e9ac6f2c1fde621b1d2d05637ea5" Nov 02 18:47:11.696445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.696405 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw"] Nov 02 18:47:11.697028 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.697011 5647 cpu_manager.go:414] "containerMap: removing container" podUID="7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" containerName="revoke-keys" Nov 02 18:47:11.697076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.697030 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" containerName="revoke-keys" Nov 02 18:47:11.697118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.697078 5647 cpu_manager.go:414] "containerMap: removing container" podUID="df2b5cd6-c2d7-48bb-9956-e264c8679d80" containerName="revoke-keys" Nov 02 18:47:11.697118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.697084 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2b5cd6-c2d7-48bb-9956-e264c8679d80" containerName="revoke-keys" Nov 02 18:47:11.697191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.697181 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2b5cd6-c2d7-48bb-9956-e264c8679d80" containerName="revoke-keys" Nov 02 18:47:11.697232 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.697194 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" containerName="revoke-keys" Nov 02 18:47:11.700955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.700932 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.754197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.754162 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:47:11.757856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.757824 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw"] Nov 02 18:47:11.802364 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.802330 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.802552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.802397 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.802666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.802638 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjs64\" (UniqueName: \"kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.804768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.804714 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:47:11.904434 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.904392 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjs64\" (UniqueName: \"kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.904643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.904488 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.904643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.904565 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.905261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.905230 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.906965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.906945 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:11.962015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:11.961934 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjs64\" (UniqueName: \"kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64\") pod \"maas-api-revoke-sub-95cb332f-hfhcw\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:12.014864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:12.014835 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:12.393696 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:12.393664 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw"] Nov 02 18:47:12.395010 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:47:12.394979 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode87e0c87_72ea_4da0_ad0a_ac8a7cf66574.slice/crio-7482d6b69685e69b09b74798c900c5faaac7066db9c9c1e5bb0d530efd5f24dc WatchSource:0}: Error finding container 7482d6b69685e69b09b74798c900c5faaac7066db9c9c1e5bb0d530efd5f24dc: Status 404 returned error can't find the container with id 7482d6b69685e69b09b74798c900c5faaac7066db9c9c1e5bb0d530efd5f24dc Nov 02 18:47:13.345377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.345330 5647 generic.go:356] "Generic (PLEG): container finished" podID="e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" containerID="5fd54c2ff903365aaa8603fe1a699a1792579b008d4299ec74e90bd121666639" exitCode=0 Nov 02 18:47:13.345880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.345411 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" event={"ID":"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574","Type":"ContainerDied","Data":"5fd54c2ff903365aaa8603fe1a699a1792579b008d4299ec74e90bd121666639"} Nov 02 18:47:13.345880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.345477 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" event={"ID":"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574","Type":"ContainerStarted","Data":"7482d6b69685e69b09b74798c900c5faaac7066db9c9c1e5bb0d530efd5f24dc"} Nov 02 18:47:13.347457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.347433 5647 generic.go:356] "Generic (PLEG): container finished" podID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerID="b88b2cbe1ed82a372772084d43b350b3814ffb77e77e1622d2e65e55d80a3151" exitCode=0 Nov 02 18:47:13.347580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.347501 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" event={"ID":"61f904cb-7a91-46e9-9b1b-64be91044bd9","Type":"ContainerDied","Data":"b88b2cbe1ed82a372772084d43b350b3814ffb77e77e1622d2e65e55d80a3151"} Nov 02 18:47:13.347580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.347552 5647 scope.go:122] "RemoveContainer" containerID="f14dd11c4a166a2f3bd9bc5e5c12d09950202c5c39fc538b439337de343fedca" Nov 02 18:47:13.386419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.386393 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:47:13.422686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422657 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.422902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422703 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.422902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422766 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.422902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422812 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.422902 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422868 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.423146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422923 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.423146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422986 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:13.423146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.423000 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:13.423146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.422999 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.423370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.423180 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.423370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.423228 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:13.423370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.423305 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket\") pod \"61f904cb-7a91-46e9-9b1b-64be91044bd9\" (UID: \"61f904cb-7a91-46e9-9b1b-64be91044bd9\") " Nov 02 18:47:13.423533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.423442 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:13.423779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.423704 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:13.424111 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.424085 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.424221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.424118 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.424221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.424138 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.424221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.424155 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.424221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.424173 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61f904cb-7a91-46e9-9b1b-64be91044bd9-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.425161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.425131 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:13.425353 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.425326 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:47:13.425446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.425424 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "kube-api-access-2mv5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:13.425446 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.425433 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token" pod "61f904cb-7a91-46e9-9b1b-64be91044bd9" (UID: "61f904cb-7a91-46e9-9b1b-64be91044bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:13.525927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.525888 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.525927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.525918 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.525927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.525927 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61f904cb-7a91-46e9-9b1b-64be91044bd9-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:13.525927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:13.525938 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mv5q\" (UniqueName: \"kubernetes.io/projected/61f904cb-7a91-46e9-9b1b-64be91044bd9-kube-api-access-2mv5q\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:14.356941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.356907 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" Nov 02 18:47:14.357434 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.356942 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh" event={"ID":"61f904cb-7a91-46e9-9b1b-64be91044bd9","Type":"ContainerDied","Data":"4321296ea7bf86716d01446acc33165fc556361519cb97173b254a4d615b4d33"} Nov 02 18:47:14.357434 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.357009 5647 scope.go:122] "RemoveContainer" containerID="b88b2cbe1ed82a372772084d43b350b3814ffb77e77e1622d2e65e55d80a3151" Nov 02 18:47:14.503022 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.502996 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:14.538227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.538192 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca\") pod \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " Nov 02 18:47:14.538415 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.538304 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64\" (UniqueName: \"kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64\") pod \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " Nov 02 18:47:14.538505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.538414 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca\") pod \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\" (UID: \"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574\") " Nov 02 18:47:14.538784 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.538756 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca" pod "e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" (UID: "e87e0c87-72ea-4da0-ad0a-ac8a7cf66574"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:14.538974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.538953 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:14.540548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.540510 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca" pod "e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" (UID: "e87e0c87-72ea-4da0-ad0a-ac8a7cf66574"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:47:14.540677 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.540650 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64" pod "e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" (UID: "e87e0c87-72ea-4da0-ad0a-ac8a7cf66574"). InnerVolumeSpecName "kube-api-access-sjs64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:14.640754 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.640652 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:14.640754 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.640688 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjs64\" (UniqueName: \"kubernetes.io/projected/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574-kube-api-access-sjs64\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:14.759588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.759550 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh"] Nov 02 18:47:14.762301 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:14.762270 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8f40154e-openshift-default-7fdfc657fc-9qvgh"] Nov 02 18:47:15.081492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:15.081416 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" path="/var/lib/kubelet/pods/61f904cb-7a91-46e9-9b1b-64be91044bd9/volumes" Nov 02 18:47:15.366200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:15.366169 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" Nov 02 18:47:15.366606 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:15.366209 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw" event={"ID":"e87e0c87-72ea-4da0-ad0a-ac8a7cf66574","Type":"ContainerDied","Data":"7482d6b69685e69b09b74798c900c5faaac7066db9c9c1e5bb0d530efd5f24dc"} Nov 02 18:47:15.366606 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:15.366261 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7482d6b69685e69b09b74798c900c5faaac7066db9c9c1e5bb0d530efd5f24dc" Nov 02 18:47:15.960134 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:15.960095 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x"] Nov 02 18:47:16.064474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:16.064440 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz"] Nov 02 18:47:20.960718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:20.960649 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerName="payload-processing" containerID="cri-o://88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8" gracePeriod=25 Nov 02 18:47:21.065261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:21.065214 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerName="payload-pre-processing" containerID="cri-o://a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a" gracePeriod=25 Nov 02 18:47:30.824877 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:30.824834 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:47:30.825403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:30.824912 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:47:35.948245 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:35.948206 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:47:38.187122 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:38.187091 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq"] Nov 02 18:47:38.187535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:38.187290 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" containerID="cri-o://38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d" gracePeriod=30 Nov 02 18:47:38.595412 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:38.595378 5647 patch_prober.go:29] interesting pod/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:47:38.595584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:38.595439 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:47:40.824327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:40.824283 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:47:40.824327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:40.824317 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:47:41.331856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.331778 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:47:41.363897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.363866 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume\") pod \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " Nov 02 18:47:41.364085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.364053 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6\" (UniqueName: \"kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6\") pod \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\" (UID: \"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731\") " Nov 02 18:47:41.364334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.364289 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume" pod "fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" (UID: "fa6e4cd1-24fe-400c-b01d-4f7b6a83e731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:41.364658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.364531 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:41.367059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.367032 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6" pod "fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" (UID: "fa6e4cd1-24fe-400c-b01d-4f7b6a83e731"). InnerVolumeSpecName "kube-api-access-kkfh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:41.382769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.381353 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:47:41.465155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.465117 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume\") pod \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " Nov 02 18:47:41.465365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.465305 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn\" (UniqueName: \"kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn\") pod \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\" (UID: \"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5\") " Nov 02 18:47:41.465553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.465526 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume" pod "c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" (UID: "c87d3f30-6f61-4eea-b2da-ea6b9ac665b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:41.465866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.465844 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:41.465944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.465873 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkfh6\" (UniqueName: \"kubernetes.io/projected/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731-kube-api-access-kkfh6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:41.467859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.467821 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn" pod "c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" (UID: "c87d3f30-6f61-4eea-b2da-ea6b9ac665b5"). InnerVolumeSpecName "kube-api-access-gg2mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:41.568328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.568249 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg2mn\" (UniqueName: \"kubernetes.io/projected/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5-kube-api-access-gg2mn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:41.602625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.602587 5647 generic.go:356] "Generic (PLEG): container finished" podID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerID="88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8" exitCode=0 Nov 02 18:47:41.602625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.602646 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" Nov 02 18:47:41.602625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.602670 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" event={"ID":"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731","Type":"ContainerDied","Data":"88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8"} Nov 02 18:47:41.602625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.602760 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x" event={"ID":"fa6e4cd1-24fe-400c-b01d-4f7b6a83e731","Type":"ContainerDied","Data":"4ffe6017b3606d0b22e8e84dafca7afdb4817b7d85301867b2785a5cb35c02a2"} Nov 02 18:47:41.602625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.602762 5647 scope.go:122] "RemoveContainer" containerID="88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8" Nov 02 18:47:41.606947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.606913 5647 generic.go:356] "Generic (PLEG): container finished" podID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerID="a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a" exitCode=0 Nov 02 18:47:41.607101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.607023 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" event={"ID":"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5","Type":"ContainerDied","Data":"a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a"} Nov 02 18:47:41.607101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.607069 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" event={"ID":"c87d3f30-6f61-4eea-b2da-ea6b9ac665b5","Type":"ContainerDied","Data":"87c95f415fd286b66dfacaac3ddbb9f7ee48e4b4c5ff45689219c762edd35489"} Nov 02 18:47:41.607249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.607154 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz" Nov 02 18:47:41.616278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.616255 5647 scope.go:122] "RemoveContainer" containerID="88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8" Nov 02 18:47:41.616632 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:41.616604 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8\": container with ID starting with 88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8 not found: ID does not exist" containerID="88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8" Nov 02 18:47:41.616706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.616647 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8"} err="failed to get container status \"88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8\": rpc error: code = NotFound desc = could not find container \"88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8\": container with ID starting with 88f3a87a5126ed768fd46c851d41c7cd6c3ebd73da0260f3a0a05359f88edce8 not found: ID does not exist" Nov 02 18:47:41.616706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.616674 5647 scope.go:122] "RemoveContainer" containerID="a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a" Nov 02 18:47:41.626152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.626134 5647 scope.go:122] "RemoveContainer" containerID="a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a" Nov 02 18:47:41.626491 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:41.626460 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a\": container with ID starting with a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a not found: ID does not exist" containerID="a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a" Nov 02 18:47:41.626590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.626502 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a"} err="failed to get container status \"a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a\": rpc error: code = NotFound desc = could not find container \"a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a\": container with ID starting with a01d41182c56631824690e0d7161be49d03119d4dbd56725207338a8479d781a not found: ID does not exist" Nov 02 18:47:41.770884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.770835 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz"] Nov 02 18:47:41.773549 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.773519 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-dd92ac-868bf8b9bc-6gpdz"] Nov 02 18:47:41.882410 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.882320 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x"] Nov 02 18:47:41.887586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:41.887554 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-dd92ac-856775db45-54q5x"] Nov 02 18:47:43.082001 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.081970 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" path="/var/lib/kubelet/pods/c87d3f30-6f61-4eea-b2da-ea6b9ac665b5/volumes" Nov 02 18:47:43.082407 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.082377 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" path="/var/lib/kubelet/pods/fa6e4cd1-24fe-400c-b01d-4f7b6a83e731/volumes" Nov 02 18:47:43.461995 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.461966 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:47:43.491510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491478 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491537 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491615 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491666 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491701 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491763 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491807 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.491871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491839 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:43.492588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491887 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.492588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491926 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket\") pod \"b75529ed-454c-417f-8bc5-4b6bf3840a72\" (UID: \"b75529ed-454c-417f-8bc5-4b6bf3840a72\") " Nov 02 18:47:43.492588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.491935 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:47:43.492588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492126 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:43.492588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492152 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:43.492588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492338 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:43.492891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492594 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.492891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492619 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b75529ed-454c-417f-8bc5-4b6bf3840a72-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.492891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492636 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.492891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492651 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.492891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.492671 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.494896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.494799 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:47:43.495025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.494923 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:47:43.495025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.494934 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:43.495025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.494980 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz" pod "b75529ed-454c-417f-8bc5-4b6bf3840a72" (UID: "b75529ed-454c-417f-8bc5-4b6bf3840a72"). InnerVolumeSpecName "kube-api-access-c4kkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:47:43.594145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.594107 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4kkz\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-kube-api-access-c4kkz\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.594145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.594138 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.594145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.594149 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.594145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.594158 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b75529ed-454c-417f-8bc5-4b6bf3840a72-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:47:43.624673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.624574 5647 generic.go:356] "Generic (PLEG): container finished" podID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerID="38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d" exitCode=0 Nov 02 18:47:43.624673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.624625 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" Nov 02 18:47:43.624673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.624657 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" event={"ID":"b75529ed-454c-417f-8bc5-4b6bf3840a72","Type":"ContainerDied","Data":"38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d"} Nov 02 18:47:43.624962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.624707 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq" event={"ID":"b75529ed-454c-417f-8bc5-4b6bf3840a72","Type":"ContainerDied","Data":"8766d4970615bb948d556de42cfd8887bac030aaecd4fbf044ddd42cda032ccf"} Nov 02 18:47:43.624962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.624744 5647 scope.go:122] "RemoveContainer" containerID="38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d" Nov 02 18:47:43.636188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.636166 5647 scope.go:122] "RemoveContainer" containerID="7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226" Nov 02 18:47:43.645850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.645830 5647 scope.go:122] "RemoveContainer" containerID="38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d" Nov 02 18:47:43.646151 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:43.646133 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d\": container with ID starting with 38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d not found: ID does not exist" containerID="38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d" Nov 02 18:47:43.646202 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.646162 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d"} err="failed to get container status \"38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d\": rpc error: code = NotFound desc = could not find container \"38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d\": container with ID starting with 38b3b254ecf8c7a4229feee516cd36a85dce31859355eb74b2ab1234f01f6c3d not found: ID does not exist" Nov 02 18:47:43.646202 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.646181 5647 scope.go:122] "RemoveContainer" containerID="7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226" Nov 02 18:47:43.646479 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:47:43.646454 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226\": container with ID starting with 7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226 not found: ID does not exist" containerID="7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226" Nov 02 18:47:43.646516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.646489 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226"} err="failed to get container status \"7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226\": rpc error: code = NotFound desc = could not find container \"7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226\": container with ID starting with 7123d2cb51d445fc62ff14ee052ce9fc64328024897f0bb9a116e40808364226 not found: ID does not exist" Nov 02 18:47:43.799080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.799044 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq"] Nov 02 18:47:43.804576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:43.804539 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-134555-openshift-default-69bb45c76f-9khzq"] Nov 02 18:47:45.080970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:47:45.080931 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" path="/var/lib/kubelet/pods/b75529ed-454c-417f-8bc5-4b6bf3840a72/volumes" Nov 02 18:48:05.947364 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:05.947326 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:48:08.201905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:08.201866 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:48:17.226943 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:17.226910 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff"] Nov 02 18:48:17.337524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:17.337487 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl"] Nov 02 18:48:22.228231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:22.228177 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerName="payload-processing" containerID="cri-o://345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2" gracePeriod=25 Nov 02 18:48:22.338688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:22.338643 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerName="payload-pre-processing" containerID="cri-o://8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64" gracePeriod=25 Nov 02 18:48:23.372279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.372241 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d"] Nov 02 18:48:23.372759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.372450 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" podUID="fb595908-9b1b-4a55-802e-56b20a8f1ee9" containerName="maas-api" containerID="cri-o://09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526" gracePeriod=30 Nov 02 18:48:23.672199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.672166 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:48:23.770227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.756133 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs\") pod \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " Nov 02 18:48:23.770227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.756360 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca\") pod \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " Nov 02 18:48:23.770227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.756594 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd\" (UniqueName: \"kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd\") pod \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " Nov 02 18:48:23.770227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.756680 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\") pod \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\" (UID: \"fb595908-9b1b-4a55-802e-56b20a8f1ee9\") " Nov 02 18:48:23.770227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.759131 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca" pod "fb595908-9b1b-4a55-802e-56b20a8f1ee9" (UID: "fb595908-9b1b-4a55-802e-56b20a8f1ee9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:23.774150 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.774092 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls" pod "fb595908-9b1b-4a55-802e-56b20a8f1ee9" (UID: "fb595908-9b1b-4a55-802e-56b20a8f1ee9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:48:23.774324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.774144 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd" pod "fb595908-9b1b-4a55-802e-56b20a8f1ee9" (UID: "fb595908-9b1b-4a55-802e-56b20a8f1ee9"). InnerVolumeSpecName "kube-api-access-r6bnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:23.785392 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.785320 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs" pod "fb595908-9b1b-4a55-802e-56b20a8f1ee9" (UID: "fb595908-9b1b-4a55-802e-56b20a8f1ee9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:48:23.858829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.858790 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:23.858829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.858826 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb595908-9b1b-4a55-802e-56b20a8f1ee9-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:23.859075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.858844 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6bnd\" (UniqueName: \"kubernetes.io/projected/fb595908-9b1b-4a55-802e-56b20a8f1ee9-kube-api-access-r6bnd\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:23.859075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.858858 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb595908-9b1b-4a55-802e-56b20a8f1ee9-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:23.880505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.880468 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:48:23.880682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.880524 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:48:23.929270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.929168 5647 generic.go:356] "Generic (PLEG): container finished" podID="fb595908-9b1b-4a55-802e-56b20a8f1ee9" containerID="09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526" exitCode=0 Nov 02 18:48:23.929270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.929222 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" Nov 02 18:48:23.929523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.929262 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" event={"ID":"fb595908-9b1b-4a55-802e-56b20a8f1ee9","Type":"ContainerDied","Data":"09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526"} Nov 02 18:48:23.929523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.929321 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d" event={"ID":"fb595908-9b1b-4a55-802e-56b20a8f1ee9","Type":"ContainerDied","Data":"966171179d039524e9cb8a96d828759fc5b841e99c074d8428cd33f16523926e"} Nov 02 18:48:23.929523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.929353 5647 scope.go:122] "RemoveContainer" containerID="09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526" Nov 02 18:48:23.941186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.941168 5647 scope.go:122] "RemoveContainer" containerID="09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526" Nov 02 18:48:23.941519 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:23.941499 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526\": container with ID starting with 09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526 not found: ID does not exist" containerID="09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526" Nov 02 18:48:23.941615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:23.941525 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526"} err="failed to get container status \"09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526\": rpc error: code = NotFound desc = could not find container \"09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526\": container with ID starting with 09f5920c2c252d8c5a74c8d2b78e30dc2dd014e44fed96f696861f959406a526 not found: ID does not exist" Nov 02 18:48:24.098053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:24.098014 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d"] Nov 02 18:48:24.103661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:24.103632 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-dd92ac-6cb7b95f5f-sgp4d"] Nov 02 18:48:25.081311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:25.081280 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb595908-9b1b-4a55-802e-56b20a8f1ee9" path="/var/lib/kubelet/pods/fb595908-9b1b-4a55-802e-56b20a8f1ee9/volumes" Nov 02 18:48:29.719746 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:29.719708 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd"] Nov 02 18:48:29.720209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:29.719952 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" containerID="cri-o://a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee" gracePeriod=30 Nov 02 18:48:33.500403 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:33.500364 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:48:33.500815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:33.500440 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:48:33.879845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:33.879795 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:48:33.880045 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:33.879854 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:48:34.919899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:34.919870 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:48:34.929528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:34.929502 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:48:34.994894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:34.994828 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:48:35.001274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.001244 5647 generic.go:356] "Generic (PLEG): container finished" podID="59bd3539-3194-4879-87b9-48a316cda7eb" containerID="a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee" exitCode=0 Nov 02 18:48:35.001408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.001291 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" Nov 02 18:48:35.001408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.001321 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" event={"ID":"59bd3539-3194-4879-87b9-48a316cda7eb","Type":"ContainerDied","Data":"a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee"} Nov 02 18:48:35.001491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.001417 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd" event={"ID":"59bd3539-3194-4879-87b9-48a316cda7eb","Type":"ContainerDied","Data":"768264854e5d47da61e03f3e25b6b88cfc5321aa8b92b096885dfb92e4070f2e"} Nov 02 18:48:35.001491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.001451 5647 scope.go:122] "RemoveContainer" containerID="a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee" Nov 02 18:48:35.013802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.013775 5647 scope.go:122] "RemoveContainer" containerID="e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14" Nov 02 18:48:35.023127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.023107 5647 scope.go:122] "RemoveContainer" containerID="a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee" Nov 02 18:48:35.023424 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:35.023404 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee\": container with ID starting with a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee not found: ID does not exist" containerID="a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee" Nov 02 18:48:35.023479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.023441 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee"} err="failed to get container status \"a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee\": rpc error: code = NotFound desc = could not find container \"a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee\": container with ID starting with a4fb4cda1df2240c41232b110332b466c7f091c53032aea329bdb9ff736967ee not found: ID does not exist" Nov 02 18:48:35.023479 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.023463 5647 scope.go:122] "RemoveContainer" containerID="e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14" Nov 02 18:48:35.023724 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:35.023701 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14\": container with ID starting with e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14 not found: ID does not exist" containerID="e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14" Nov 02 18:48:35.023872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.023748 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14"} err="failed to get container status \"e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14\": rpc error: code = NotFound desc = could not find container \"e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14\": container with ID starting with e4896d0f1234aab4c504b64dcb09b6c889c5a44ec7cb10d68c02fc4e2d479e14 not found: ID does not exist" Nov 02 18:48:35.030252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030225 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030287 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030332 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030393 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030429 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030517 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030500 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030525 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:35.030690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030534 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030613 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:35.030690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030630 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030690 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030669 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket\") pod \"59bd3539-3194-4879-87b9-48a316cda7eb\" (UID: \"59bd3539-3194-4879-87b9-48a316cda7eb\") " Nov 02 18:48:35.030993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030803 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:35.030993 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.030837 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:35.031097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.031047 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:35.031215 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.031196 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59bd3539-3194-4879-87b9-48a316cda7eb-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.031279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.031222 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.031279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.031236 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.031279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.031250 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.031279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.031264 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.032776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.032755 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:48:35.032906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.032885 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "kube-api-access-cpnk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:35.033334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.033312 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:35.033334 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.033322 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy" pod "59bd3539-3194-4879-87b9-48a316cda7eb" (UID: "59bd3539-3194-4879-87b9-48a316cda7eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:35.132292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.132255 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59bd3539-3194-4879-87b9-48a316cda7eb-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.132292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.132285 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpnk2\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-kube-api-access-cpnk2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.132292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.132295 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59bd3539-3194-4879-87b9-48a316cda7eb-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.132292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.132305 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59bd3539-3194-4879-87b9-48a316cda7eb-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:35.440366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.440299 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd"] Nov 02 18:48:35.444310 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:35.444283 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-dd92ac-openshift-default-6d4f9d8bc9-z7mfd"] Nov 02 18:48:35.948575 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:35.948537 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:48:37.081106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:37.081070 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" path="/var/lib/kubelet/pods/59bd3539-3194-4879-87b9-48a316cda7eb/volumes" Nov 02 18:48:37.764247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:37.764212 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5"] Nov 02 18:48:37.870676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:37.870638 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2"] Nov 02 18:48:42.489825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.489698 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:48:42.554172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.554135 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl\" (UniqueName: \"kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl\") pod \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " Nov 02 18:48:42.554383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.554295 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume\") pod \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\" (UID: \"2eee8e39-18f1-443b-b577-b1f991f1e7c6\") " Nov 02 18:48:42.554797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.554712 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume" pod "2eee8e39-18f1-443b-b577-b1f991f1e7c6" (UID: "2eee8e39-18f1-443b-b577-b1f991f1e7c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:42.556638 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.556610 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl" pod "2eee8e39-18f1-443b-b577-b1f991f1e7c6" (UID: "2eee8e39-18f1-443b-b577-b1f991f1e7c6"). InnerVolumeSpecName "kube-api-access-hgzsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:42.629666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.629633 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:48:42.655940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.655904 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume\") pod \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " Nov 02 18:48:42.656188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.656168 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v\" (UniqueName: \"kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v\") pod \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\" (UID: \"5b1d380e-5259-42c2-b9b9-ddc39ec1a280\") " Nov 02 18:48:42.656360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.656334 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume" pod "5b1d380e-5259-42c2-b9b9-ddc39ec1a280" (UID: "5b1d380e-5259-42c2-b9b9-ddc39ec1a280"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:42.656578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.656562 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2eee8e39-18f1-443b-b577-b1f991f1e7c6-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:42.656644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.656580 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:42.656644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.656590 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgzsl\" (UniqueName: \"kubernetes.io/projected/2eee8e39-18f1-443b-b577-b1f991f1e7c6-kube-api-access-hgzsl\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:42.658432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.658404 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v" pod "5b1d380e-5259-42c2-b9b9-ddc39ec1a280" (UID: "5b1d380e-5259-42c2-b9b9-ddc39ec1a280"). InnerVolumeSpecName "kube-api-access-zzj4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:42.757954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.757877 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzj4v\" (UniqueName: \"kubernetes.io/projected/5b1d380e-5259-42c2-b9b9-ddc39ec1a280-kube-api-access-zzj4v\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:42.765481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.765419 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerName="payload-processing" containerID="cri-o://70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4" gracePeriod=25 Nov 02 18:48:42.872020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:42.871951 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerName="payload-pre-processing" containerID="cri-o://cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b" gracePeriod=25 Nov 02 18:48:43.053359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.053258 5647 generic.go:356] "Generic (PLEG): container finished" podID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerID="345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2" exitCode=0 Nov 02 18:48:43.053359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.053305 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" Nov 02 18:48:43.053359 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.053339 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" event={"ID":"2eee8e39-18f1-443b-b577-b1f991f1e7c6","Type":"ContainerDied","Data":"345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2"} Nov 02 18:48:43.053680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.053394 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff" event={"ID":"2eee8e39-18f1-443b-b577-b1f991f1e7c6","Type":"ContainerDied","Data":"827020a664521195e2116fb669a4fab998ad2171a776b4129b7fcce7f6108151"} Nov 02 18:48:43.053680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.053416 5647 scope.go:122] "RemoveContainer" containerID="345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2" Nov 02 18:48:43.055054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.055028 5647 generic.go:356] "Generic (PLEG): container finished" podID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerID="8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64" exitCode=0 Nov 02 18:48:43.055188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.055072 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" Nov 02 18:48:43.055188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.055117 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" event={"ID":"5b1d380e-5259-42c2-b9b9-ddc39ec1a280","Type":"ContainerDied","Data":"8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64"} Nov 02 18:48:43.055188 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.055184 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl" event={"ID":"5b1d380e-5259-42c2-b9b9-ddc39ec1a280","Type":"ContainerDied","Data":"aa299532f06d09c0da6523da7e0cf9cffb2ca1deb392e05a47b391ca9502e1ea"} Nov 02 18:48:43.064580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.064503 5647 scope.go:122] "RemoveContainer" containerID="345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2" Nov 02 18:48:43.064918 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:43.064895 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2\": container with ID starting with 345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2 not found: ID does not exist" containerID="345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2" Nov 02 18:48:43.065038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.064926 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2"} err="failed to get container status \"345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2\": rpc error: code = NotFound desc = could not find container \"345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2\": container with ID starting with 345b7b8f6618e6b1873cbb294090e8665ab28f63a51c305bc3cf934e8e98f7b2 not found: ID does not exist" Nov 02 18:48:43.065038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.064943 5647 scope.go:122] "RemoveContainer" containerID="8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64" Nov 02 18:48:43.074260 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.074241 5647 scope.go:122] "RemoveContainer" containerID="8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64" Nov 02 18:48:43.074556 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:43.074538 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64\": container with ID starting with 8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64 not found: ID does not exist" containerID="8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64" Nov 02 18:48:43.074608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.074564 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64"} err="failed to get container status \"8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64\": rpc error: code = NotFound desc = could not find container \"8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64\": container with ID starting with 8a39f6333887f50f47a55145976f1ef3516fda757678f2ca51b1971538f50b64 not found: ID does not exist" Nov 02 18:48:43.224300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.224258 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff"] Nov 02 18:48:43.274759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.274698 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-580a21-c8b5dc995-ggpff"] Nov 02 18:48:43.338961 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.338924 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl"] Nov 02 18:48:43.341626 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:43.341602 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-580a21-6b9c7755b7-gmvnl"] Nov 02 18:48:45.095424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:45.095379 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" path="/var/lib/kubelet/pods/2eee8e39-18f1-443b-b577-b1f991f1e7c6/volumes" Nov 02 18:48:45.096052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:45.096030 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" path="/var/lib/kubelet/pods/5b1d380e-5259-42c2-b9b9-ddc39ec1a280/volumes" Nov 02 18:48:45.919034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:45.918998 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss"] Nov 02 18:48:45.919257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:45.919201 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" podUID="25931f24-3219-4adb-a7bf-062e058b5ea1" containerName="maas-api" containerID="cri-o://48869a7356ce8a2d755f6542837669162ec3f9ed51cd4495b1f1231f8b32d953" gracePeriod=30 Nov 02 18:48:46.080604 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.080567 5647 generic.go:356] "Generic (PLEG): container finished" podID="25931f24-3219-4adb-a7bf-062e058b5ea1" containerID="48869a7356ce8a2d755f6542837669162ec3f9ed51cd4495b1f1231f8b32d953" exitCode=0 Nov 02 18:48:46.080833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.080637 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" event={"ID":"25931f24-3219-4adb-a7bf-062e058b5ea1","Type":"ContainerDied","Data":"48869a7356ce8a2d755f6542837669162ec3f9ed51cd4495b1f1231f8b32d953"} Nov 02 18:48:46.212078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.212043 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:48:46.302825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.301601 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\") pod \"25931f24-3219-4adb-a7bf-062e058b5ea1\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " Nov 02 18:48:46.302825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.301659 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca\") pod \"25931f24-3219-4adb-a7bf-062e058b5ea1\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " Nov 02 18:48:46.302825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.301727 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk\" (UniqueName: \"kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk\") pod \"25931f24-3219-4adb-a7bf-062e058b5ea1\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " Nov 02 18:48:46.302825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.301778 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs\") pod \"25931f24-3219-4adb-a7bf-062e058b5ea1\" (UID: \"25931f24-3219-4adb-a7bf-062e058b5ea1\") " Nov 02 18:48:46.312759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.304017 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca" pod "25931f24-3219-4adb-a7bf-062e058b5ea1" (UID: "25931f24-3219-4adb-a7bf-062e058b5ea1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:46.312759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.309325 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk" pod "25931f24-3219-4adb-a7bf-062e058b5ea1" (UID: "25931f24-3219-4adb-a7bf-062e058b5ea1"). InnerVolumeSpecName "kube-api-access-k2rlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:46.312759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.310631 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs" pod "25931f24-3219-4adb-a7bf-062e058b5ea1" (UID: "25931f24-3219-4adb-a7bf-062e058b5ea1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:48:46.328799 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.322928 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls" pod "25931f24-3219-4adb-a7bf-062e058b5ea1" (UID: "25931f24-3219-4adb-a7bf-062e058b5ea1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:48:46.403183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.403144 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:46.403183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.403179 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25931f24-3219-4adb-a7bf-062e058b5ea1-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:46.403183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.403191 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2rlk\" (UniqueName: \"kubernetes.io/projected/25931f24-3219-4adb-a7bf-062e058b5ea1-kube-api-access-k2rlk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:46.403447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:46.403200 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25931f24-3219-4adb-a7bf-062e058b5ea1-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:47.090036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:47.090009 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" Nov 02 18:48:47.090245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:47.090006 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss" event={"ID":"25931f24-3219-4adb-a7bf-062e058b5ea1","Type":"ContainerDied","Data":"6ce60460bb469ad7c10d1613de3c2c472c9921695d853abcef9b175e8fa69efd"} Nov 02 18:48:47.090245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:47.090144 5647 scope.go:122] "RemoveContainer" containerID="48869a7356ce8a2d755f6542837669162ec3f9ed51cd4495b1f1231f8b32d953" Nov 02 18:48:47.256174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:47.256096 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss"] Nov 02 18:48:47.306643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:47.306608 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-580a21-86b4695554-hgnss"] Nov 02 18:48:49.081116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:49.081081 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25931f24-3219-4adb-a7bf-062e058b5ea1" path="/var/lib/kubelet/pods/25931f24-3219-4adb-a7bf-062e058b5ea1/volumes" Nov 02 18:48:51.262546 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:51.262513 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn"] Nov 02 18:48:51.262986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:51.262805 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" containerID="cri-o://0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf" gracePeriod=30 Nov 02 18:48:51.325231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:51.325200 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:48:51.356059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:51.356017 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:48:51.356288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:51.356031 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:48:52.317528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.317483 5647 patch_prober.go:29] interesting pod/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:48:52.317966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.317559 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:48:52.368470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368432 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw"] Nov 02 18:48:52.368955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368942 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerName="payload-pre-processing" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368957 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerName="payload-pre-processing" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368969 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" containerName="revoke-keys" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368976 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" containerName="revoke-keys" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368982 5647 cpu_manager.go:414] "containerMap: removing container" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368987 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.368997 5647 cpu_manager.go:414] "containerMap: removing container" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" Nov 02 18:48:52.369010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369003 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369018 5647 cpu_manager.go:414] "containerMap: removing container" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerName="payload-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369023 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerName="payload-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369032 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369037 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369045 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369050 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369062 5647 cpu_manager.go:414] "containerMap: removing container" podUID="25931f24-3219-4adb-a7bf-062e058b5ea1" containerName="maas-api" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369066 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25931f24-3219-4adb-a7bf-062e058b5ea1" containerName="maas-api" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369078 5647 cpu_manager.go:414] "containerMap: removing container" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369083 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369097 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerName="payload-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369101 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerName="payload-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369108 5647 cpu_manager.go:414] "containerMap: removing container" podUID="fb595908-9b1b-4a55-802e-56b20a8f1ee9" containerName="maas-api" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369113 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb595908-9b1b-4a55-802e-56b20a8f1ee9" containerName="maas-api" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369124 5647 cpu_manager.go:414] "containerMap: removing container" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerName="payload-pre-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369129 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerName="payload-pre-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369202 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369211 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="25931f24-3219-4adb-a7bf-062e058b5ea1" containerName="maas-api" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369221 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369228 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eee8e39-18f1-443b-b577-b1f991f1e7c6" containerName="payload-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369233 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb595908-9b1b-4a55-802e-56b20a8f1ee9" containerName="maas-api" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369241 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b1d380e-5259-42c2-b9b9-ddc39ec1a280" containerName="payload-pre-processing" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369247 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" Nov 02 18:48:52.369243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369254 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87d3f30-6f61-4eea-b2da-ea6b9ac665b5" containerName="payload-pre-processing" Nov 02 18:48:52.370008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369259 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" containerName="revoke-keys" Nov 02 18:48:52.370008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369264 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="59bd3539-3194-4879-87b9-48a316cda7eb" containerName="istio-proxy" Nov 02 18:48:52.370008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369269 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" Nov 02 18:48:52.370008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.369275 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6e4cd1-24fe-400c-b01d-4f7b6a83e731" containerName="payload-processing" Nov 02 18:48:52.372844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.372823 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.427706 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.427663 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:48:52.431783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.431752 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw"] Nov 02 18:48:52.478243 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.478202 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:48:52.488912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.488881 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.489110 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.488962 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rl5q\" (UniqueName: \"kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.489110 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.489083 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.614776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.598720 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.614776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.598851 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rl5q\" (UniqueName: \"kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.614776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.598921 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.614776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.600153 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.620651 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.620615 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.658558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.658517 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rl5q\" (UniqueName: \"kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q\") pod \"maas-api-revoke-sub-a69ce2e2-hwcdw\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:52.686794 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:52.686762 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:53.070549 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:53.070519 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw"] Nov 02 18:48:53.071548 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:48:53.071514 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca210fb_e2b1_4d5f_81aa_4e63ba7451d1.slice/crio-d9a627d769e0e5ade058b489a42108f6a8a7ea2e536ab934f7162f039f51499c WatchSource:0}: Error finding container d9a627d769e0e5ade058b489a42108f6a8a7ea2e536ab934f7162f039f51499c: Status 404 returned error can't find the container with id d9a627d769e0e5ade058b489a42108f6a8a7ea2e536ab934f7162f039f51499c Nov 02 18:48:53.129399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:53.129365 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" event={"ID":"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1","Type":"ContainerStarted","Data":"d9a627d769e0e5ade058b489a42108f6a8a7ea2e536ab934f7162f039f51499c"} Nov 02 18:48:54.137351 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:54.137317 5647 generic.go:356] "Generic (PLEG): container finished" podID="9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" containerID="44e6acf698c9cf80e5453532771585fc29c6f3ab57a3f43f6203d74957bd13c1" exitCode=0 Nov 02 18:48:54.137727 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:54.137406 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" event={"ID":"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1","Type":"ContainerDied","Data":"44e6acf698c9cf80e5453532771585fc29c6f3ab57a3f43f6203d74957bd13c1"} Nov 02 18:48:55.291308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.291281 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:55.433007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.432915 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca\") pod \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " Nov 02 18:48:55.433007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.432954 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca\") pod \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " Nov 02 18:48:55.433257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.433065 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q\" (UniqueName: \"kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q\") pod \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\" (UID: \"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1\") " Nov 02 18:48:55.433544 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.433505 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca" pod "9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" (UID: "9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:55.435428 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.435404 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca" pod "9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" (UID: "9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:48:55.436159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.436137 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q" pod "9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" (UID: "9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1"). InnerVolumeSpecName "kube-api-access-8rl5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:55.537252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.537215 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:55.537624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.537599 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:55.537724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:55.537630 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rl5q\" (UniqueName: \"kubernetes.io/projected/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1-kube-api-access-8rl5q\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.150108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.150069 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" event={"ID":"9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1","Type":"ContainerDied","Data":"d9a627d769e0e5ade058b489a42108f6a8a7ea2e536ab934f7162f039f51499c"} Nov 02 18:48:56.150306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.150125 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9a627d769e0e5ade058b489a42108f6a8a7ea2e536ab934f7162f039f51499c" Nov 02 18:48:56.150306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.150081 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw" Nov 02 18:48:56.529320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.529297 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:48:56.650402 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650365 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650624 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650577 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650630 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650629 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:56.650700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650680 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650722 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650795 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650857 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.650896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650870 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:56.650896 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650894 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.651179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.650924 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token\") pod \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\" (UID: \"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0\") " Nov 02 18:48:56.651179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651104 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651354 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651481 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651503 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651517 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651532 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.651982 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.653418 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.653582 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "kube-api-access-5d4hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.654055 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:48:56.654776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.654473 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo" pod "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" (UID: "aecb6c0b-afc3-46a0-8f03-3acd652fbfe0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:48:56.753239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.753153 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.753239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.753186 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.753239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.753196 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.753239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.753205 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4hm\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-kube-api-access-5d4hm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:56.753239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:56.753215 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:48:57.158203 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.158172 5647 generic.go:356] "Generic (PLEG): container finished" podID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerID="0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf" exitCode=0 Nov 02 18:48:57.158365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.158235 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" Nov 02 18:48:57.158365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.158257 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" event={"ID":"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0","Type":"ContainerDied","Data":"0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf"} Nov 02 18:48:57.158365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.158317 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn" event={"ID":"aecb6c0b-afc3-46a0-8f03-3acd652fbfe0","Type":"ContainerDied","Data":"33117dd955497b3b5c67277b0fef7748036fc2e99df4bc3d77c3d166b3e7ba9c"} Nov 02 18:48:57.158365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.158345 5647 scope.go:122] "RemoveContainer" containerID="0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf" Nov 02 18:48:57.170008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.169985 5647 scope.go:122] "RemoveContainer" containerID="fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b" Nov 02 18:48:57.181151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.181129 5647 scope.go:122] "RemoveContainer" containerID="0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf" Nov 02 18:48:57.181524 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:57.181504 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf\": container with ID starting with 0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf not found: ID does not exist" containerID="0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf" Nov 02 18:48:57.181593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.181533 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf"} err="failed to get container status \"0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf\": rpc error: code = NotFound desc = could not find container \"0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf\": container with ID starting with 0c1521c81e8fcb47fedc1384ef65cfcad71f2702caf6bc3d7241af54bd8fefdf not found: ID does not exist" Nov 02 18:48:57.181593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.181555 5647 scope.go:122] "RemoveContainer" containerID="fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b" Nov 02 18:48:57.181918 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:48:57.181896 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b\": container with ID starting with fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b not found: ID does not exist" containerID="fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b" Nov 02 18:48:57.182025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.181923 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b"} err="failed to get container status \"fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b\": rpc error: code = NotFound desc = could not find container \"fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b\": container with ID starting with fff218bad053d70d570941879d7b66a0ff65d9f3b508d298ea14fd4d0ff5787b not found: ID does not exist" Nov 02 18:48:57.325220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.325133 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn"] Nov 02 18:48:57.376169 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.376088 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4fe24d-openshift-default-5b65dd9cd7-9pndn"] Nov 02 18:48:57.458818 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.458774 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp"] Nov 02 18:48:57.459064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:57.459035 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" containerID="cri-o://c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4" gracePeriod=30 Nov 02 18:48:59.082076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:48:59.082042 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" path="/var/lib/kubelet/pods/aecb6c0b-afc3-46a0-8f03-3acd652fbfe0/volumes" Nov 02 18:49:01.355524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:01.355476 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:49:01.356048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:01.355491 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:49:02.748153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.748128 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:49:02.835204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835173 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835217 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835292 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835314 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835348 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835406 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835457 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835583 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835597 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:02.835646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835635 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs\") pod \"54da1589-034c-48a5-b9dc-b10a59c90b76\" (UID: \"54da1589-034c-48a5-b9dc-b10a59c90b76\") " Nov 02 18:49:02.835974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835628 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:02.835974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835822 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:02.835974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835855 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:02.835974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.835927 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:02.836419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.836394 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.836492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.836425 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54da1589-034c-48a5-b9dc-b10a59c90b76-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.836492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.836441 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.836492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.836457 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.836492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.836472 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.837712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.837683 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:02.837712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.837697 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:02.837712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.837688 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:49:02.838047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.837989 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f" pod "54da1589-034c-48a5-b9dc-b10a59c90b76" (UID: "54da1589-034c-48a5-b9dc-b10a59c90b76"). InnerVolumeSpecName "kube-api-access-k5c4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:02.937829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.937794 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.937829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.937829 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.938014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.937844 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54da1589-034c-48a5-b9dc-b10a59c90b76-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:02.938014 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:02.937860 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5c4f\" (UniqueName: \"kubernetes.io/projected/54da1589-034c-48a5-b9dc-b10a59c90b76-kube-api-access-k5c4f\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:03.019299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.019268 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:49:03.106283 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.106262 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:49:03.138857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.138827 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc\" (UniqueName: \"kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc\") pod \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " Nov 02 18:49:03.139069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.138912 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume\") pod \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\" (UID: \"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc\") " Nov 02 18:49:03.139069 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.138987 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume\") pod \"dd21626f-0d9e-43db-9cbc-953645494d5f\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " Nov 02 18:49:03.139207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.139072 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64\" (UniqueName: \"kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64\") pod \"dd21626f-0d9e-43db-9cbc-953645494d5f\" (UID: \"dd21626f-0d9e-43db-9cbc-953645494d5f\") " Nov 02 18:49:03.140102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.140004 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume" pod "dd21626f-0d9e-43db-9cbc-953645494d5f" (UID: "dd21626f-0d9e-43db-9cbc-953645494d5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:03.140762 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.140721 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume" pod "be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" (UID: "be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:03.142525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.142502 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64" pod "dd21626f-0d9e-43db-9cbc-953645494d5f" (UID: "dd21626f-0d9e-43db-9cbc-953645494d5f"). InnerVolumeSpecName "kube-api-access-sqz64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:03.142639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.142539 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc" pod "be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" (UID: "be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc"). InnerVolumeSpecName "kube-api-access-w6vdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:03.202335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.202295 5647 generic.go:356] "Generic (PLEG): container finished" podID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerID="70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4" exitCode=0 Nov 02 18:49:03.202532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.202348 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" Nov 02 18:49:03.202532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.202376 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" event={"ID":"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc","Type":"ContainerDied","Data":"70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4"} Nov 02 18:49:03.202532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.202435 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5" event={"ID":"be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc","Type":"ContainerDied","Data":"86878bfd0a278fcd90106439d91ccfc1b854257c0c353b0fd2f1d9b93d8e6cfb"} Nov 02 18:49:03.202532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.202462 5647 scope.go:122] "RemoveContainer" containerID="70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4" Nov 02 18:49:03.204506 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.204477 5647 generic.go:356] "Generic (PLEG): container finished" podID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerID="c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4" exitCode=0 Nov 02 18:49:03.204670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.204517 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" Nov 02 18:49:03.204670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.204556 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" event={"ID":"54da1589-034c-48a5-b9dc-b10a59c90b76","Type":"ContainerDied","Data":"c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4"} Nov 02 18:49:03.204670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.204599 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp" event={"ID":"54da1589-034c-48a5-b9dc-b10a59c90b76","Type":"ContainerDied","Data":"8bf18757013ea3428813c55c5927bba684ac02835c11bfc0f3930ff48391a680"} Nov 02 18:49:03.206279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.206252 5647 generic.go:356] "Generic (PLEG): container finished" podID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerID="cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b" exitCode=0 Nov 02 18:49:03.206279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.206280 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" Nov 02 18:49:03.206476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.206335 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" event={"ID":"dd21626f-0d9e-43db-9cbc-953645494d5f","Type":"ContainerDied","Data":"cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b"} Nov 02 18:49:03.206476 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.206386 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2" event={"ID":"dd21626f-0d9e-43db-9cbc-953645494d5f","Type":"ContainerDied","Data":"4e38d64653e04cef5a514c26ba26664b270879c559cc4493a95d2544fda0b9e8"} Nov 02 18:49:03.214182 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.214162 5647 scope.go:122] "RemoveContainer" containerID="70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4" Nov 02 18:49:03.214499 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:03.214474 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4\": container with ID starting with 70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4 not found: ID does not exist" containerID="70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4" Nov 02 18:49:03.214577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.214507 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4"} err="failed to get container status \"70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4\": rpc error: code = NotFound desc = could not find container \"70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4\": container with ID starting with 70253b968258e60b7d3f76c72eaeb35760f833970154f102ed651229670f08d4 not found: ID does not exist" Nov 02 18:49:03.214577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.214526 5647 scope.go:122] "RemoveContainer" containerID="c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4" Nov 02 18:49:03.227049 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.227024 5647 scope.go:122] "RemoveContainer" containerID="466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5" Nov 02 18:49:03.236635 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.236617 5647 scope.go:122] "RemoveContainer" containerID="c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4" Nov 02 18:49:03.236947 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:03.236928 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4\": container with ID starting with c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4 not found: ID does not exist" containerID="c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4" Nov 02 18:49:03.237023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.236957 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4"} err="failed to get container status \"c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4\": rpc error: code = NotFound desc = could not find container \"c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4\": container with ID starting with c0a3188ec18c4aad768e57350fecec4011664a9eec74bb23c01e18097994ffa4 not found: ID does not exist" Nov 02 18:49:03.237023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.236980 5647 scope.go:122] "RemoveContainer" containerID="466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5" Nov 02 18:49:03.237241 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:03.237221 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5\": container with ID starting with 466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5 not found: ID does not exist" containerID="466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5" Nov 02 18:49:03.237241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.237243 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5"} err="failed to get container status \"466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5\": rpc error: code = NotFound desc = could not find container \"466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5\": container with ID starting with 466ae6a146c326a0383fc2089f33a0cf4a3e6a869f95c45066a0c93bd0b748e5 not found: ID does not exist" Nov 02 18:49:03.237387 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.237256 5647 scope.go:122] "RemoveContainer" containerID="cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b" Nov 02 18:49:03.240490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.240465 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:03.240581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.240498 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd21626f-0d9e-43db-9cbc-953645494d5f-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:03.240581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.240516 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqz64\" (UniqueName: \"kubernetes.io/projected/dd21626f-0d9e-43db-9cbc-953645494d5f-kube-api-access-sqz64\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:03.240581 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.240532 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6vdc\" (UniqueName: \"kubernetes.io/projected/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc-kube-api-access-w6vdc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:03.246197 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.246181 5647 scope.go:122] "RemoveContainer" containerID="cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b" Nov 02 18:49:03.246474 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:03.246457 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b\": container with ID starting with cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b not found: ID does not exist" containerID="cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b" Nov 02 18:49:03.246523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.246480 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b"} err="failed to get container status \"cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b\": rpc error: code = NotFound desc = could not find container \"cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b\": container with ID starting with cd7b1caa0b8475ab16527997a0827c739f81dd34c70a20ec1e041c252aa7de4b not found: ID does not exist" Nov 02 18:49:03.424631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.424598 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2"] Nov 02 18:49:03.426925 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.426898 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-4ecad8-5fcbc4dc7f-whpb2"] Nov 02 18:49:03.537778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.537672 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp"] Nov 02 18:49:03.540545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.540520 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-580a21-openshift-default-5646686b8f-2ddqp"] Nov 02 18:49:03.651939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.651886 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5"] Nov 02 18:49:03.654060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:03.654040 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-4ecad8-7847d67bb7-s4mt5"] Nov 02 18:49:05.080615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:05.080578 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" path="/var/lib/kubelet/pods/54da1589-034c-48a5-b9dc-b10a59c90b76/volumes" Nov 02 18:49:05.081042 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:05.081026 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" path="/var/lib/kubelet/pods/be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc/volumes" Nov 02 18:49:05.081357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:05.081343 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" path="/var/lib/kubelet/pods/dd21626f-0d9e-43db-9cbc-953645494d5f/volumes" Nov 02 18:49:05.947071 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:05.947031 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:49:09.187363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187332 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz"] Nov 02 18:49:09.187903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187817 5647 cpu_manager.go:414] "containerMap: removing container" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" Nov 02 18:49:09.187903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187832 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" Nov 02 18:49:09.187903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187849 5647 cpu_manager.go:414] "containerMap: removing container" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" Nov 02 18:49:09.187903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187857 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f904cb-7a91-46e9-9b1b-64be91044bd9" containerName="istio-proxy" Nov 02 18:49:09.187903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187884 5647 cpu_manager.go:414] "containerMap: removing container" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" Nov 02 18:49:09.187903 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187889 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187915 5647 cpu_manager.go:414] "containerMap: removing container" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerName="payload-processing" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187925 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerName="payload-processing" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187941 5647 cpu_manager.go:414] "containerMap: removing container" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerName="payload-pre-processing" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187946 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerName="payload-pre-processing" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187954 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" containerName="revoke-keys" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.187958 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" containerName="revoke-keys" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188034 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75529ed-454c-417f-8bc5-4b6bf3840a72" containerName="istio-proxy" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188047 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188057 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188063 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="be81d0ac-2dd1-4e66-bfc0-c89ecc5ed1cc" containerName="payload-processing" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188070 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd21626f-0d9e-43db-9cbc-953645494d5f" containerName="payload-pre-processing" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188077 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" containerName="revoke-keys" Nov 02 18:49:09.188143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.188086 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" Nov 02 18:49:09.192891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.192870 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.296846 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.296814 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:49:09.297052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.296927 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:49:09.299255 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.299235 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz"] Nov 02 18:49:09.313295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.313264 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc88w\" (UniqueName: \"kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.313443 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.313311 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.313443 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.313406 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.414301 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.414264 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.414484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.414405 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc88w\" (UniqueName: \"kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.414484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.414446 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.415148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.415129 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.416836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.416817 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.472290 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.472214 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc88w\" (UniqueName: \"kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w\") pod \"maas-api-revoke-sub-da24afb7-94xnz\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.506749 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.506708 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:09.893470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:09.893439 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz"] Nov 02 18:49:09.894840 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:49:09.894810 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be5d81b_ba54_46fd_9c5e_e40ec23f9aa2.slice/crio-32c2970b2c9fe8c6f4a561a39f534c908aa9f6a6e8eeced3bba70862d98239f0 WatchSource:0}: Error finding container 32c2970b2c9fe8c6f4a561a39f534c908aa9f6a6e8eeced3bba70862d98239f0: Status 404 returned error can't find the container with id 32c2970b2c9fe8c6f4a561a39f534c908aa9f6a6e8eeced3bba70862d98239f0 Nov 02 18:49:10.254071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:10.254036 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" event={"ID":"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2","Type":"ContainerStarted","Data":"58e0b847cf0b40453830aec3e941c6abdbe7b253709d03b6e31ffa5dc84b5f17"} Nov 02 18:49:10.254453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:10.254101 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" event={"ID":"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2","Type":"ContainerStarted","Data":"32c2970b2c9fe8c6f4a561a39f534c908aa9f6a6e8eeced3bba70862d98239f0"} Nov 02 18:49:10.369357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:10.369299 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" podStartSLOduration=1.369282611 podStartE2EDuration="1.369282611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:49:10.368048339 +0000 UTC m=+5137.441575543" watchObservedRunningTime="2026-11-02 18:49:10.369282611 +0000 UTC m=+5137.442809843" Nov 02 18:49:11.266876 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:11.266842 5647 generic.go:356] "Generic (PLEG): container finished" podID="8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" containerID="58e0b847cf0b40453830aec3e941c6abdbe7b253709d03b6e31ffa5dc84b5f17" exitCode=0 Nov 02 18:49:11.267309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:11.266891 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" event={"ID":"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2","Type":"ContainerDied","Data":"58e0b847cf0b40453830aec3e941c6abdbe7b253709d03b6e31ffa5dc84b5f17"} Nov 02 18:49:12.442221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.442191 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:12.549866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.549770 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca\") pod \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " Nov 02 18:49:12.549866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.549860 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca\") pod \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " Nov 02 18:49:12.550076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.550044 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w\" (UniqueName: \"kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w\") pod \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\" (UID: \"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2\") " Nov 02 18:49:12.550254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.550227 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca" pod "8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" (UID: "8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:12.550421 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.550402 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:12.552195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.552164 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca" pod "8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" (UID: "8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:12.552338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.552250 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w" pod "8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" (UID: "8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2"). InnerVolumeSpecName "kube-api-access-tc88w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:12.651319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.651275 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc88w\" (UniqueName: \"kubernetes.io/projected/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-kube-api-access-tc88w\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:12.651319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:12.651312 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:13.283269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:13.283239 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:49:13.283269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:13.283253 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" event={"ID":"8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2","Type":"ContainerDied","Data":"32c2970b2c9fe8c6f4a561a39f534c908aa9f6a6e8eeced3bba70862d98239f0"} Nov 02 18:49:13.283510 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:13.283306 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c2970b2c9fe8c6f4a561a39f534c908aa9f6a6e8eeced3bba70862d98239f0" Nov 02 18:49:14.967442 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:14.967403 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d"] Nov 02 18:49:14.967900 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:14.967673 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" containerID="cri-o://6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2" gracePeriod=30 Nov 02 18:49:17.749432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:17.749381 5647 patch_prober.go:29] interesting pod/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:49:17.749890 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:17.749447 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:49:20.233075 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.233014 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:49:20.335771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.335708 5647 generic.go:356] "Generic (PLEG): container finished" podID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerID="6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2" exitCode=0 Nov 02 18:49:20.335771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.335775 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" Nov 02 18:49:20.336023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.335795 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" event={"ID":"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9","Type":"ContainerDied","Data":"6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2"} Nov 02 18:49:20.336023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.335848 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d" event={"ID":"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9","Type":"ContainerDied","Data":"72dcf08f78a944f71d8313c83988e36a821cd494def13a7869b3f0346acd5b62"} Nov 02 18:49:20.336023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.335870 5647 scope.go:122] "RemoveContainer" containerID="6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2" Nov 02 18:49:20.339239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339216 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339323 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339416 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339366 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339465 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339444 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339486 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339490 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:20.339600 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339528 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339600 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339572 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339640 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339725 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339703 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:20.339861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339721 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token\") pod \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\" (UID: \"72c963b7-64e0-437c-ade6-6f0c4f7fb3c9\") " Nov 02 18:49:20.339995 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.339974 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:20.340196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.340137 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:20.340196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.340155 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.340196 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.340178 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.340557 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.340534 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:20.341927 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.341901 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "kube-api-access-d94cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:20.342210 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.342182 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:49:20.342400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.342382 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:20.342462 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.342423 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy" pod "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" (UID: "72c963b7-64e0-437c-ade6-6f0c4f7fb3c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:20.346840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.346817 5647 scope.go:122] "RemoveContainer" containerID="8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3" Nov 02 18:49:20.367947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.367924 5647 scope.go:122] "RemoveContainer" containerID="6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2" Nov 02 18:49:20.368319 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:20.368293 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2\": container with ID starting with 6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2 not found: ID does not exist" containerID="6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2" Nov 02 18:49:20.368384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.368332 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2"} err="failed to get container status \"6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2\": rpc error: code = NotFound desc = could not find container \"6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2\": container with ID starting with 6ad3092bece98204501d8d01931c436f1f2e687e2a2e7e5c2052ca69fd64cef2 not found: ID does not exist" Nov 02 18:49:20.368384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.368351 5647 scope.go:122] "RemoveContainer" containerID="8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3" Nov 02 18:49:20.368642 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:20.368624 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3\": container with ID starting with 8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3 not found: ID does not exist" containerID="8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3" Nov 02 18:49:20.368687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.368645 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3"} err="failed to get container status \"8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3\": rpc error: code = NotFound desc = could not find container \"8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3\": container with ID starting with 8a9016d65592a5515c5d86fab904c5dd50765e1121888e44f82a0bb0c0727fc3 not found: ID does not exist" Nov 02 18:49:20.441538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441506 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.441538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441536 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.441826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441549 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d94cg\" (UniqueName: \"kubernetes.io/projected/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-kube-api-access-d94cg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.441826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441561 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.441826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441570 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.441826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441581 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.441826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.441589 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:20.757391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.757348 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d"] Nov 02 18:49:20.760918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:20.760890 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-413a4e-openshift-default-746b7d6d5f-kxj6d"] Nov 02 18:49:21.080575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:21.080493 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" path="/var/lib/kubelet/pods/72c963b7-64e0-437c-ade6-6f0c4f7fb3c9/volumes" Nov 02 18:49:34.839939 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.839894 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp"] Nov 02 18:49:34.840424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840405 5647 cpu_manager.go:414] "containerMap: removing container" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" Nov 02 18:49:34.840474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840428 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecb6c0b-afc3-46a0-8f03-3acd652fbfe0" containerName="istio-proxy" Nov 02 18:49:34.840474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840454 5647 cpu_manager.go:414] "containerMap: removing container" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" Nov 02 18:49:34.840474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840460 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" Nov 02 18:49:34.840578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840489 5647 cpu_manager.go:414] "containerMap: removing container" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" Nov 02 18:49:34.840578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840495 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" Nov 02 18:49:34.840578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840536 5647 cpu_manager.go:414] "containerMap: removing container" podUID="8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" containerName="revoke-keys" Nov 02 18:49:34.840578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840541 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" containerName="revoke-keys" Nov 02 18:49:34.840708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840610 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="54da1589-034c-48a5-b9dc-b10a59c90b76" containerName="istio-proxy" Nov 02 18:49:34.840708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840620 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" Nov 02 18:49:34.840708 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.840630 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" containerName="revoke-keys" Nov 02 18:49:34.845297 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.845275 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:34.913138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.913060 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp"] Nov 02 18:49:34.921082 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.921041 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:34.921258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.921188 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7jtq\" (UniqueName: \"kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:34.921258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.921236 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:34.954941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.954908 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:49:34.955172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:34.954997 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:49:35.022131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.022098 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.022329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.022219 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.022329 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.022274 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7jtq\" (UniqueName: \"kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.022881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.022860 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.024681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.024658 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.084228 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.084000 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7jtq\" (UniqueName: \"kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq\") pod \"maas-api-revoke-sub-650d02a1-8bkfp\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.159104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.159010 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:35.531365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:35.531232 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp"] Nov 02 18:49:35.534482 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:49:35.534457 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6693ff_12fc_4ce5_a0cc_53609f012ee0.slice/crio-d1caffe1d41e1419601d28d8c43cc038322154dd22ac8000e73d2372625a02a4 WatchSource:0}: Error finding container d1caffe1d41e1419601d28d8c43cc038322154dd22ac8000e73d2372625a02a4: Status 404 returned error can't find the container with id d1caffe1d41e1419601d28d8c43cc038322154dd22ac8000e73d2372625a02a4 Nov 02 18:49:35.946379 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:35.946341 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:49:36.443264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:36.443227 5647 generic.go:356] "Generic (PLEG): container finished" podID="ed6693ff-12fc-4ce5-a0cc-53609f012ee0" containerID="d83ab2536310139a9ce227b641a91d57ff5654aa3ac40ea0934a3766363e608b" exitCode=0 Nov 02 18:49:36.443435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:36.443257 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" event={"ID":"ed6693ff-12fc-4ce5-a0cc-53609f012ee0","Type":"ContainerDied","Data":"d83ab2536310139a9ce227b641a91d57ff5654aa3ac40ea0934a3766363e608b"} Nov 02 18:49:36.443435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:36.443330 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" event={"ID":"ed6693ff-12fc-4ce5-a0cc-53609f012ee0","Type":"ContainerStarted","Data":"d1caffe1d41e1419601d28d8c43cc038322154dd22ac8000e73d2372625a02a4"} Nov 02 18:49:37.603169 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.603144 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:37.654807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.654776 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq\" (UniqueName: \"kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq\") pod \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " Nov 02 18:49:37.654964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.654876 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca\") pod \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " Nov 02 18:49:37.654964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.654910 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca\") pod \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\" (UID: \"ed6693ff-12fc-4ce5-a0cc-53609f012ee0\") " Nov 02 18:49:37.655305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.655280 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca" pod "ed6693ff-12fc-4ce5-a0cc-53609f012ee0" (UID: "ed6693ff-12fc-4ce5-a0cc-53609f012ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:37.657193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.657167 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca" pod "ed6693ff-12fc-4ce5-a0cc-53609f012ee0" (UID: "ed6693ff-12fc-4ce5-a0cc-53609f012ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:37.657270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.657172 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq" pod "ed6693ff-12fc-4ce5-a0cc-53609f012ee0" (UID: "ed6693ff-12fc-4ce5-a0cc-53609f012ee0"). InnerVolumeSpecName "kube-api-access-h7jtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:37.756723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.756693 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:37.756723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.756727 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:37.756966 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:37.756775 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7jtq\" (UniqueName: \"kubernetes.io/projected/ed6693ff-12fc-4ce5-a0cc-53609f012ee0-kube-api-access-h7jtq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:38.456528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:38.456498 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" Nov 02 18:49:38.456767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:38.456533 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp" event={"ID":"ed6693ff-12fc-4ce5-a0cc-53609f012ee0","Type":"ContainerDied","Data":"d1caffe1d41e1419601d28d8c43cc038322154dd22ac8000e73d2372625a02a4"} Nov 02 18:49:38.456767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:38.456591 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1caffe1d41e1419601d28d8c43cc038322154dd22ac8000e73d2372625a02a4" Nov 02 18:49:39.216007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:39.215975 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg"] Nov 02 18:49:46.097322 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.097284 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5"] Nov 02 18:49:46.097914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.097540 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" podUID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" containerName="maas-api" containerID="cri-o://07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e" gracePeriod=30 Nov 02 18:49:46.184305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.184255 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" podUID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.230:8443/health\": dial tcp 10.134.0.230:8443: connect: connection refused" Nov 02 18:49:46.382278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.382245 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:49:46.475664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.475531 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\") pod \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " Nov 02 18:49:46.475664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.475577 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs\") pod \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " Nov 02 18:49:46.475983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.475703 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca\") pod \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " Nov 02 18:49:46.475983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.475793 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88\" (UniqueName: \"kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88\") pod \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\" (UID: \"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6\") " Nov 02 18:49:46.477440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.477411 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca" pod "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" (UID: "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:46.479335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.479295 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls" pod "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" (UID: "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:46.479845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.479814 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88" pod "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" (UID: "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6"). InnerVolumeSpecName "kube-api-access-plz88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:46.480138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.480112 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs" pod "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" (UID: "46cb0afe-e0df-46b0-abce-a4dcb62b0fe6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:46.510662 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.510629 5647 generic.go:356] "Generic (PLEG): container finished" podID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" containerID="07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e" exitCode=0 Nov 02 18:49:46.510894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.510774 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" event={"ID":"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6","Type":"ContainerDied","Data":"07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e"} Nov 02 18:49:46.510894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.510788 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" Nov 02 18:49:46.510894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.510852 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5" event={"ID":"46cb0afe-e0df-46b0-abce-a4dcb62b0fe6","Type":"ContainerDied","Data":"74dcb3a40783d3deef6691118331d44806a5a1a665fc737e1ee9e30dac3c211f"} Nov 02 18:49:46.510894 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.510885 5647 scope.go:122] "RemoveContainer" containerID="07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e" Nov 02 18:49:46.522401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.522383 5647 scope.go:122] "RemoveContainer" containerID="07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e" Nov 02 18:49:46.522721 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:46.522696 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e\": container with ID starting with 07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e not found: ID does not exist" containerID="07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e" Nov 02 18:49:46.522806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.522758 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e"} err="failed to get container status \"07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e\": rpc error: code = NotFound desc = could not find container \"07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e\": container with ID starting with 07a83dd11288d6ad89750e04bdb094b4911bb76d0827cf7a177f215447e1c25e not found: ID does not exist" Nov 02 18:49:46.577836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.577788 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:46.577836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.577828 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:46.577836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.577842 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:46.578078 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.577855 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plz88\" (UniqueName: \"kubernetes.io/projected/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6-kube-api-access-plz88\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:46.678327 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.678239 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5"] Nov 02 18:49:46.680470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:46.680447 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0ad26a-6467f8dd-kcts5"] Nov 02 18:49:47.081276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:47.081195 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" path="/var/lib/kubelet/pods/46cb0afe-e0df-46b0-abce-a4dcb62b0fe6/volumes" Nov 02 18:49:49.595343 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.595296 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j"] Nov 02 18:49:49.596023 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596006 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ed6693ff-12fc-4ce5-a0cc-53609f012ee0" containerName="revoke-keys" Nov 02 18:49:49.596088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596026 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6693ff-12fc-4ce5-a0cc-53609f012ee0" containerName="revoke-keys" Nov 02 18:49:49.596088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596052 5647 cpu_manager.go:414] "containerMap: removing container" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" Nov 02 18:49:49.596088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596060 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" Nov 02 18:49:49.596189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596110 5647 cpu_manager.go:414] "containerMap: removing container" podUID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" containerName="maas-api" Nov 02 18:49:49.596189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596119 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" containerName="maas-api" Nov 02 18:49:49.596256 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596204 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6693ff-12fc-4ce5-a0cc-53609f012ee0" containerName="revoke-keys" Nov 02 18:49:49.596256 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596220 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c963b7-64e0-437c-ade6-6f0c4f7fb3c9" containerName="istio-proxy" Nov 02 18:49:49.596256 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.596233 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cb0afe-e0df-46b0-abce-a4dcb62b0fe6" containerName="maas-api" Nov 02 18:49:49.601161 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.601142 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.653664 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.653630 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j"] Nov 02 18:49:49.701222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.701183 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:49:49.701222 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.701205 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:49:49.708710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.708686 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwc9\" (UniqueName: \"kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.708891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.708873 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.708933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.708912 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.810274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.810240 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwc9\" (UniqueName: \"kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.810484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.810386 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.810484 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.810433 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.811102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.811070 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.813015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.812988 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.868670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.868593 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwc9\" (UniqueName: \"kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9\") pod \"maas-api-revoke-sub-80c63887-s5d4j\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:49.913678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:49.913644 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:50.288199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:50.288161 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j"] Nov 02 18:49:50.289813 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:49:50.289786 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e8691b_3138_4ea4_975f_3a341eeb9c3e.slice/crio-c2055776261f0930803e2684413c7bd3d9c2c0cc042d4aa9ebccd8d8085d31f1 WatchSource:0}: Error finding container c2055776261f0930803e2684413c7bd3d9c2c0cc042d4aa9ebccd8d8085d31f1: Status 404 returned error can't find the container with id c2055776261f0930803e2684413c7bd3d9c2c0cc042d4aa9ebccd8d8085d31f1 Nov 02 18:49:50.537938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:50.537853 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" event={"ID":"64e8691b-3138-4ea4-975f-3a341eeb9c3e","Type":"ContainerStarted","Data":"c2055776261f0930803e2684413c7bd3d9c2c0cc042d4aa9ebccd8d8085d31f1"} Nov 02 18:49:51.545175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:51.545079 5647 generic.go:356] "Generic (PLEG): container finished" podID="64e8691b-3138-4ea4-975f-3a341eeb9c3e" containerID="8eb225324d41c83680bcd7d47960104e069db11e11005f932b5ad9853f0c3240" exitCode=0 Nov 02 18:49:51.545175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:51.545158 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" event={"ID":"64e8691b-3138-4ea4-975f-3a341eeb9c3e","Type":"ContainerDied","Data":"8eb225324d41c83680bcd7d47960104e069db11e11005f932b5ad9853f0c3240"} Nov 02 18:49:52.684943 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.684912 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:52.848490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.848453 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9\" (UniqueName: \"kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9\") pod \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " Nov 02 18:49:52.848729 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.848508 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca\") pod \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " Nov 02 18:49:52.848729 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.848547 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca\") pod \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\" (UID: \"64e8691b-3138-4ea4-975f-3a341eeb9c3e\") " Nov 02 18:49:52.848905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.848880 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca" pod "64e8691b-3138-4ea4-975f-3a341eeb9c3e" (UID: "64e8691b-3138-4ea4-975f-3a341eeb9c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:52.849130 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.849113 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:52.850950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.850914 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca" pod "64e8691b-3138-4ea4-975f-3a341eeb9c3e" (UID: "64e8691b-3138-4ea4-975f-3a341eeb9c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:52.850950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.850938 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9" pod "64e8691b-3138-4ea4-975f-3a341eeb9c3e" (UID: "64e8691b-3138-4ea4-975f-3a341eeb9c3e"). InnerVolumeSpecName "kube-api-access-jcwc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:52.949975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.949936 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcwc9\" (UniqueName: \"kubernetes.io/projected/64e8691b-3138-4ea4-975f-3a341eeb9c3e-kube-api-access-jcwc9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:52.949975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:52.949970 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64e8691b-3138-4ea4-975f-3a341eeb9c3e-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:53.557924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:53.557885 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" Nov 02 18:49:53.558128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:53.557891 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j" event={"ID":"64e8691b-3138-4ea4-975f-3a341eeb9c3e","Type":"ContainerDied","Data":"c2055776261f0930803e2684413c7bd3d9c2c0cc042d4aa9ebccd8d8085d31f1"} Nov 02 18:49:53.558128 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:53.558011 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2055776261f0930803e2684413c7bd3d9c2c0cc042d4aa9ebccd8d8085d31f1" Nov 02 18:49:54.219805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.219769 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:49:54.237588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.237536 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" containerID="cri-o://c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4" gracePeriod=45 Nov 02 18:49:54.276518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.276477 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused" Nov 02 18:49:54.501436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.501404 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 18:49:54.566537 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.566503 5647 generic.go:356] "Generic (PLEG): container finished" podID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerID="c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4" exitCode=0 Nov 02 18:49:54.566766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.566542 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" Nov 02 18:49:54.566766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.566583 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" event={"ID":"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d","Type":"ContainerDied","Data":"c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4"} Nov 02 18:49:54.566766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.566633 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg" event={"ID":"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d","Type":"ContainerDied","Data":"aeaa4b91405a258dd198b1406a9d7b562d8b057e9523d40dbfa2bde7397bbcfc"} Nov 02 18:49:54.566766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.566650 5647 scope.go:122] "RemoveContainer" containerID="c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4" Nov 02 18:49:54.576398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.576378 5647 scope.go:122] "RemoveContainer" containerID="c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4" Nov 02 18:49:54.576790 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:49:54.576763 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4\": container with ID starting with c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4 not found: ID does not exist" containerID="c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4" Nov 02 18:49:54.576881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.576804 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4"} err="failed to get container status \"c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4\": rpc error: code = NotFound desc = could not find container \"c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4\": container with ID starting with c22075f3f8fdc1f63a3b04d79abedfda667fe9d0648bc3931ac04c17616b3dc4 not found: ID does not exist" Nov 02 18:49:54.670279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670245 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh\" (UniqueName: \"kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh\") pod \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " Nov 02 18:49:54.670488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670305 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache\") pod \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " Nov 02 18:49:54.670488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670341 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm\") pod \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " Nov 02 18:49:54.670488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670392 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home\") pod \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " Nov 02 18:49:54.670687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670507 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs\") pod \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " Nov 02 18:49:54.670687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670560 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir\") pod \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\" (UID: \"5b4c3c62-35ca-4a21-88fe-b850c7f63f6d\") " Nov 02 18:49:54.670687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670641 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache" pod "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" (UID: "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:54.671029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.670981 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir" pod "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" (UID: "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:54.671142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.671045 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:54.671142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.671126 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home" pod "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" (UID: "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:54.675528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.674392 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs" pod "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" (UID: "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:54.675528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.674409 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh" pod "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" (UID: "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d"). InnerVolumeSpecName "kube-api-access-h7mlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:54.675528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.675483 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm" pod "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" (UID: "5b4c3c62-35ca-4a21-88fe-b850c7f63f6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:49:54.790712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.787550 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:54.790712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.787592 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:54.790712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.787610 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:54.790712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.787625 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7mlh\" (UniqueName: \"kubernetes.io/projected/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-kube-api-access-h7mlh\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:54.790712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.787640 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:54.992897 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:54.992834 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg"] Nov 02 18:49:55.043306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:55.043221 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dc5689/distinct-2-w1-dc5689-kserve-bf79c9554-v88zg"] Nov 02 18:49:55.080543 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:55.080507 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" path="/var/lib/kubelet/pods/5b4c3c62-35ca-4a21-88fe-b850c7f63f6d/volumes" Nov 02 18:49:55.519436 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:55.519395 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4"] Nov 02 18:49:55.520106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:55.519671 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" containerID="cri-o://bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6" gracePeriod=30 Nov 02 18:49:56.600052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600015 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd"] Nov 02 18:49:56.600545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600480 5647 cpu_manager.go:414] "containerMap: removing container" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" Nov 02 18:49:56.600545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600491 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" Nov 02 18:49:56.600545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600529 5647 cpu_manager.go:414] "containerMap: removing container" podUID="64e8691b-3138-4ea4-975f-3a341eeb9c3e" containerName="revoke-keys" Nov 02 18:49:56.600545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600535 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e8691b-3138-4ea4-975f-3a341eeb9c3e" containerName="revoke-keys" Nov 02 18:49:56.600694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600599 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e8691b-3138-4ea4-975f-3a341eeb9c3e" containerName="revoke-keys" Nov 02 18:49:56.600694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.600608 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b4c3c62-35ca-4a21-88fe-b850c7f63f6d" containerName="main" Nov 02 18:49:56.610595 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.610562 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.616663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.616632 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.616887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.616813 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrr8t\" (UniqueName: \"kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.616979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.616961 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.661675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.661629 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd"] Nov 02 18:49:56.708872 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.708835 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:49:56.709062 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.708870 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:49:56.718314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.718281 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.718500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.718354 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.718500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.718406 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrr8t\" (UniqueName: \"kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.718945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.718925 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.720918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.720898 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.776341 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.776306 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrr8t\" (UniqueName: \"kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t\") pod \"maas-api-revoke-sub-527e8f4e-28rfd\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:56.926080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:56.925992 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:57.300437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:57.300409 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd"] Nov 02 18:49:57.302381 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:49:57.302352 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac824068_ece7_410e_975c_6493ee530557.slice/crio-8aa374e9bcec6a5bf1ba9a36e14eeada554886c9784d9044f5d5a32556cf6469 WatchSource:0}: Error finding container 8aa374e9bcec6a5bf1ba9a36e14eeada554886c9784d9044f5d5a32556cf6469: Status 404 returned error can't find the container with id 8aa374e9bcec6a5bf1ba9a36e14eeada554886c9784d9044f5d5a32556cf6469 Nov 02 18:49:57.594758 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:57.594691 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" event={"ID":"ac824068-ece7-410e-975c-6493ee530557","Type":"ContainerStarted","Data":"8aa374e9bcec6a5bf1ba9a36e14eeada554886c9784d9044f5d5a32556cf6469"} Nov 02 18:49:58.603317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:58.603282 5647 generic.go:356] "Generic (PLEG): container finished" podID="ac824068-ece7-410e-975c-6493ee530557" containerID="0600e9d5975f8f289c140d0d0272386d97ed592fbee55a22054f1f8e6c911957" exitCode=0 Nov 02 18:49:58.603768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:58.603325 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" event={"ID":"ac824068-ece7-410e-975c-6493ee530557","Type":"ContainerDied","Data":"0600e9d5975f8f289c140d0d0272386d97ed592fbee55a22054f1f8e6c911957"} Nov 02 18:49:59.740759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.740710 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:49:59.749862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.749840 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca\") pod \"ac824068-ece7-410e-975c-6493ee530557\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " Nov 02 18:49:59.749986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.749964 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca\") pod \"ac824068-ece7-410e-975c-6493ee530557\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " Nov 02 18:49:59.750032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.750008 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t\" (UniqueName: \"kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t\") pod \"ac824068-ece7-410e-975c-6493ee530557\" (UID: \"ac824068-ece7-410e-975c-6493ee530557\") " Nov 02 18:49:59.750148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.750125 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca" pod "ac824068-ece7-410e-975c-6493ee530557" (UID: "ac824068-ece7-410e-975c-6493ee530557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:49:59.750302 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.750289 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac824068-ece7-410e-975c-6493ee530557-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:59.752229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.752204 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t" pod "ac824068-ece7-410e-975c-6493ee530557" (UID: "ac824068-ece7-410e-975c-6493ee530557"). InnerVolumeSpecName "kube-api-access-jrr8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:49:59.752317 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.752262 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca" pod "ac824068-ece7-410e-975c-6493ee530557" (UID: "ac824068-ece7-410e-975c-6493ee530557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:49:59.851905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.851870 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac824068-ece7-410e-975c-6493ee530557-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:59.851905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.851898 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrr8t\" (UniqueName: \"kubernetes.io/projected/ac824068-ece7-410e-975c-6493ee530557-kube-api-access-jrr8t\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:49:59.989848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.989721 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:49:59.989848 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:49:59.989812 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:50:00.616917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.616881 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" event={"ID":"ac824068-ece7-410e-975c-6493ee530557","Type":"ContainerDied","Data":"8aa374e9bcec6a5bf1ba9a36e14eeada554886c9784d9044f5d5a32556cf6469"} Nov 02 18:50:00.617094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.616917 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd" Nov 02 18:50:00.617094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.616933 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa374e9bcec6a5bf1ba9a36e14eeada554886c9784d9044f5d5a32556cf6469" Nov 02 18:50:00.797065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.797003 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:50:00.863354 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863326 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863388 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863416 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863448 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863467 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863486 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863551 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863603 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.863914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863809 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:00.863914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863832 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:00.864123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863950 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:00.864123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.863960 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket\") pod \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\" (UID: \"007f9dc9-1629-4cc5-9f79-5355b5c7f11f\") " Nov 02 18:50:00.864123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864081 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:00.864302 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864158 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:00.864364 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864350 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.864548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864372 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.864548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864390 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.864548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864405 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.864548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.864482 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.865917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.865892 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:00.866053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.866031 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:00.866100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.866045 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:50:00.866100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.866047 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r" pod "007f9dc9-1629-4cc5-9f79-5355b5c7f11f" (UID: "007f9dc9-1629-4cc5-9f79-5355b5c7f11f"). InnerVolumeSpecName "kube-api-access-9mw4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:00.965136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.965095 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m"] Nov 02 18:50:00.966006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.965953 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.966006 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.965987 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.966231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.966011 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:00.966231 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:00.966025 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mw4r\" (UniqueName: \"kubernetes.io/projected/007f9dc9-1629-4cc5-9f79-5355b5c7f11f-kube-api-access-9mw4r\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:01.625881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.625844 5647 generic.go:356] "Generic (PLEG): container finished" podID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerID="bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6" exitCode=0 Nov 02 18:50:01.626109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.625921 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" Nov 02 18:50:01.626109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.625925 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" event={"ID":"007f9dc9-1629-4cc5-9f79-5355b5c7f11f","Type":"ContainerDied","Data":"bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6"} Nov 02 18:50:01.626109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.625985 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4" event={"ID":"007f9dc9-1629-4cc5-9f79-5355b5c7f11f","Type":"ContainerDied","Data":"d1ac60aa524ecdd53fdccec787e0ebbbd543844ebde00d7c4468b0527765f857"} Nov 02 18:50:01.626109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.626007 5647 scope.go:122] "RemoveContainer" containerID="bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6" Nov 02 18:50:01.635983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.635964 5647 scope.go:122] "RemoveContainer" containerID="c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a" Nov 02 18:50:01.645280 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.645105 5647 scope.go:122] "RemoveContainer" containerID="bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6" Nov 02 18:50:01.645445 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:01.645421 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6\": container with ID starting with bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6 not found: ID does not exist" containerID="bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6" Nov 02 18:50:01.645491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.645456 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6"} err="failed to get container status \"bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6\": rpc error: code = NotFound desc = could not find container \"bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6\": container with ID starting with bc18faa454e22dcdc5d07e5b4366708a5b3581cc1b51e4e3eb0ab85432f91dc6 not found: ID does not exist" Nov 02 18:50:01.645491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.645477 5647 scope.go:122] "RemoveContainer" containerID="c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a" Nov 02 18:50:01.645714 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:01.645697 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a\": container with ID starting with c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a not found: ID does not exist" containerID="c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a" Nov 02 18:50:01.645859 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.645721 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a"} err="failed to get container status \"c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a\": rpc error: code = NotFound desc = could not find container \"c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a\": container with ID starting with c44c511b19a63a0aa684942d2b2a10a4165a9c2af78e5c734919dae68224a29a not found: ID does not exist" Nov 02 18:50:01.796586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.796546 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4"] Nov 02 18:50:01.847201 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:01.847168 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-0ad26a-openshift-default-6cb874b8bf-phkd4"] Nov 02 18:50:03.094258 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:03.094222 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" path="/var/lib/kubelet/pods/007f9dc9-1629-4cc5-9f79-5355b5c7f11f/volumes" Nov 02 18:50:05.947346 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:05.947306 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:50:06.340520 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.340484 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69"] Nov 02 18:50:06.340963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.340950 5647 cpu_manager.go:414] "containerMap: removing container" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" Nov 02 18:50:06.341018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.340965 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" Nov 02 18:50:06.341018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.340993 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ac824068-ece7-410e-975c-6493ee530557" containerName="revoke-keys" Nov 02 18:50:06.341018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.340999 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac824068-ece7-410e-975c-6493ee530557" containerName="revoke-keys" Nov 02 18:50:06.341123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.341070 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" Nov 02 18:50:06.341123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.341083 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac824068-ece7-410e-975c-6493ee530557" containerName="revoke-keys" Nov 02 18:50:06.341123 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.341090 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" Nov 02 18:50:06.345644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.345627 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.402661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.402625 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:50:06.404522 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.404495 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69"] Nov 02 18:50:06.426586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.426553 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.426775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.426680 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.426843 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.426819 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws258\" (UniqueName: \"kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.452267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.452242 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:50:06.529149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.529115 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.529362 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.529300 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.529461 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.529372 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws258\" (UniqueName: \"kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.530477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.530446 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.539666 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.539631 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.587239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.587212 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws258\" (UniqueName: \"kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258\") pod \"maas-api-revoke-sub-57a002c0-x4s69\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:06.658817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:06.658724 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:07.038447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:07.038421 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69"] Nov 02 18:50:07.039961 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:50:07.039925 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff654825_580e_4095_bd64_54d77c445da0.slice/crio-fecbfdae8e2cb65699cf22f2e0817a1f9622f94c10f54530327485184b85f794 WatchSource:0}: Error finding container fecbfdae8e2cb65699cf22f2e0817a1f9622f94c10f54530327485184b85f794: Status 404 returned error can't find the container with id fecbfdae8e2cb65699cf22f2e0817a1f9622f94c10f54530327485184b85f794 Nov 02 18:50:07.663121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:07.663081 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" event={"ID":"ff654825-580e-4095-bd64-54d77c445da0","Type":"ContainerStarted","Data":"e2aeef64a10dd18aa8e79d8333b8180d66b8cd87ed83d01df2a98d9e38f251ad"} Nov 02 18:50:07.663321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:07.663141 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" event={"ID":"ff654825-580e-4095-bd64-54d77c445da0","Type":"ContainerStarted","Data":"fecbfdae8e2cb65699cf22f2e0817a1f9622f94c10f54530327485184b85f794"} Nov 02 18:50:07.779968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:07.779904 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" podStartSLOduration=1.7798872540000001 podStartE2EDuration="1.779887254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:07.777623428 +0000 UTC m=+5194.851150630" watchObservedRunningTime="2026-11-02 18:50:07.779887254 +0000 UTC m=+5194.853414457" Nov 02 18:50:08.670815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:08.670776 5647 generic.go:356] "Generic (PLEG): container finished" podID="ff654825-580e-4095-bd64-54d77c445da0" containerID="e2aeef64a10dd18aa8e79d8333b8180d66b8cd87ed83d01df2a98d9e38f251ad" exitCode=0 Nov 02 18:50:08.671234 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:08.670877 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" event={"ID":"ff654825-580e-4095-bd64-54d77c445da0","Type":"ContainerDied","Data":"e2aeef64a10dd18aa8e79d8333b8180d66b8cd87ed83d01df2a98d9e38f251ad"} Nov 02 18:50:09.819149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.819122 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:09.866950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.866912 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258\" (UniqueName: \"kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258\") pod \"ff654825-580e-4095-bd64-54d77c445da0\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " Nov 02 18:50:09.867172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.866969 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca\") pod \"ff654825-580e-4095-bd64-54d77c445da0\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " Nov 02 18:50:09.867172 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.867013 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca\") pod \"ff654825-580e-4095-bd64-54d77c445da0\" (UID: \"ff654825-580e-4095-bd64-54d77c445da0\") " Nov 02 18:50:09.867425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.867393 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca" pod "ff654825-580e-4095-bd64-54d77c445da0" (UID: "ff654825-580e-4095-bd64-54d77c445da0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:09.869206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.869184 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca" pod "ff654825-580e-4095-bd64-54d77c445da0" (UID: "ff654825-580e-4095-bd64-54d77c445da0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:09.869300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.869271 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258" pod "ff654825-580e-4095-bd64-54d77c445da0" (UID: "ff654825-580e-4095-bd64-54d77c445da0"). InnerVolumeSpecName "kube-api-access-ws258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:09.968262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.968183 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws258\" (UniqueName: \"kubernetes.io/projected/ff654825-580e-4095-bd64-54d77c445da0-kube-api-access-ws258\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:09.968262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.968213 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff654825-580e-4095-bd64-54d77c445da0-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:09.968262 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:09.968224 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff654825-580e-4095-bd64-54d77c445da0-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:10.684054 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:10.684018 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" Nov 02 18:50:10.684235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:10.684061 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69" event={"ID":"ff654825-580e-4095-bd64-54d77c445da0","Type":"ContainerDied","Data":"fecbfdae8e2cb65699cf22f2e0817a1f9622f94c10f54530327485184b85f794"} Nov 02 18:50:10.684235 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:10.684115 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fecbfdae8e2cb65699cf22f2e0817a1f9622f94c10f54530327485184b85f794" Nov 02 18:50:12.068802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:12.068767 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9"] Nov 02 18:50:12.069287 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:12.068975 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" containerID="cri-o://04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e" gracePeriod=30 Nov 02 18:50:13.096131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.096082 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8"] Nov 02 18:50:13.096697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.096515 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ff654825-580e-4095-bd64-54d77c445da0" containerName="revoke-keys" Nov 02 18:50:13.096697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.096527 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff654825-580e-4095-bd64-54d77c445da0" containerName="revoke-keys" Nov 02 18:50:13.096697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.096576 5647 cpu_manager.go:414] "containerMap: removing container" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" Nov 02 18:50:13.096697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.096581 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007f9dc9-1629-4cc5-9f79-5355b5c7f11f" containerName="istio-proxy" Nov 02 18:50:13.096697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.096653 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff654825-580e-4095-bd64-54d77c445da0" containerName="revoke-keys" Nov 02 18:50:13.100229 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.100202 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.144955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.144921 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8"] Nov 02 18:50:13.192513 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.192476 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:50:13.192721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.192662 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:50:13.201289 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.201247 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.201398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.201304 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.201502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.201407 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dgl\" (UniqueName: \"kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.302523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.302487 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dgl\" (UniqueName: \"kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.302710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.302608 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.302710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.302638 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.303267 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.303246 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.305124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.305100 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.361042 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.360967 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dgl\" (UniqueName: \"kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl\") pod \"maas-api-revoke-sub-cf4edede-69cd8\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.413989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.413935 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:13.788470 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:13.788440 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8"] Nov 02 18:50:13.789454 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:50:13.789427 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f752bb_62ca_48a6_95a0_0c871d543dc6.slice/crio-fdc187a2ad15fe9fc853330a4f1ed9c4b55ed237e23fff114460d8e12861ce4c WatchSource:0}: Error finding container fdc187a2ad15fe9fc853330a4f1ed9c4b55ed237e23fff114460d8e12861ce4c: Status 404 returned error can't find the container with id fdc187a2ad15fe9fc853330a4f1ed9c4b55ed237e23fff114460d8e12861ce4c Nov 02 18:50:14.324272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:14.324238 5647 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:50:14.324631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:14.324293 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:50:14.708136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:14.708106 5647 generic.go:356] "Generic (PLEG): container finished" podID="45f752bb-62ca-48a6-95a0-0c871d543dc6" containerID="e206176227a05b32c2b6bdad412273abae57e9e9e2ad2a755c2ac6372bb4a44c" exitCode=0 Nov 02 18:50:14.708300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:14.708184 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" event={"ID":"45f752bb-62ca-48a6-95a0-0c871d543dc6","Type":"ContainerDied","Data":"e206176227a05b32c2b6bdad412273abae57e9e9e2ad2a755c2ac6372bb4a44c"} Nov 02 18:50:14.708300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:14.708235 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" event={"ID":"45f752bb-62ca-48a6-95a0-0c871d543dc6","Type":"ContainerStarted","Data":"fdc187a2ad15fe9fc853330a4f1ed9c4b55ed237e23fff114460d8e12861ce4c"} Nov 02 18:50:15.727150 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.727119 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p"] Nov 02 18:50:15.731138 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.731117 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.790579 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.790543 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p"] Nov 02 18:50:15.828603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.828561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgz89\" (UniqueName: \"kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.828875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.828618 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.828875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.828777 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.853347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.853320 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:15.929984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.929945 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl\" (UniqueName: \"kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl\") pod \"45f752bb-62ca-48a6-95a0-0c871d543dc6\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " Nov 02 18:50:15.930171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.930042 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca\") pod \"45f752bb-62ca-48a6-95a0-0c871d543dc6\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " Nov 02 18:50:15.930227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.930208 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca\") pod \"45f752bb-62ca-48a6-95a0-0c871d543dc6\" (UID: \"45f752bb-62ca-48a6-95a0-0c871d543dc6\") " Nov 02 18:50:15.930399 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.930376 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgz89\" (UniqueName: \"kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.930564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.930422 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.930564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.930471 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.930697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.930673 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca" pod "45f752bb-62ca-48a6-95a0-0c871d543dc6" (UID: "45f752bb-62ca-48a6-95a0-0c871d543dc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:15.931272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.931245 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.932627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.932597 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl" pod "45f752bb-62ca-48a6-95a0-0c871d543dc6" (UID: "45f752bb-62ca-48a6-95a0-0c871d543dc6"). InnerVolumeSpecName "kube-api-access-22dgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:15.932717 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.932705 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca" pod "45f752bb-62ca-48a6-95a0-0c871d543dc6" (UID: "45f752bb-62ca-48a6-95a0-0c871d543dc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:15.933147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.933123 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.988557 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.988484 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgz89\" (UniqueName: \"kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89\") pod \"maas-api-revoke-sub-863dfc69-zng9p\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:15.991685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:15.991645 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" containerID="cri-o://2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118" gracePeriod=45 Nov 02 18:50:16.031767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.031711 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22dgl\" (UniqueName: \"kubernetes.io/projected/45f752bb-62ca-48a6-95a0-0c871d543dc6-kube-api-access-22dgl\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.031767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.031761 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.031767 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.031772 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f752bb-62ca-48a6-95a0-0c871d543dc6-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.044481 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.044454 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:16.260338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.260310 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 18:50:16.335107 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335067 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6\" (UniqueName: \"kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6\") pod \"511048c8-8540-4625-836a-dac96d03f292\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " Nov 02 18:50:16.335300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335122 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home\") pod \"511048c8-8540-4625-836a-dac96d03f292\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " Nov 02 18:50:16.335300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335153 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs\" (UniqueName: \"kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs\") pod \"511048c8-8540-4625-836a-dac96d03f292\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " Nov 02 18:50:16.335300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335177 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir\") pod \"511048c8-8540-4625-836a-dac96d03f292\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " Nov 02 18:50:16.335456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335297 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache\") pod \"511048c8-8540-4625-836a-dac96d03f292\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " Nov 02 18:50:16.335456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335357 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm\") pod \"511048c8-8540-4625-836a-dac96d03f292\" (UID: \"511048c8-8540-4625-836a-dac96d03f292\") " Nov 02 18:50:16.335456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335441 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home" pod "511048c8-8540-4625-836a-dac96d03f292" (UID: "511048c8-8540-4625-836a-dac96d03f292"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:16.335630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335491 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir" pod "511048c8-8540-4625-836a-dac96d03f292" (UID: "511048c8-8540-4625-836a-dac96d03f292"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:16.335630 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335573 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache" pod "511048c8-8540-4625-836a-dac96d03f292" (UID: "511048c8-8540-4625-836a-dac96d03f292"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:16.335829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335807 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.335829 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335827 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.335983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.335842 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.337633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.337605 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm" pod "511048c8-8540-4625-836a-dac96d03f292" (UID: "511048c8-8540-4625-836a-dac96d03f292"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:16.337709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.337612 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6" pod "511048c8-8540-4625-836a-dac96d03f292" (UID: "511048c8-8540-4625-836a-dac96d03f292"). InnerVolumeSpecName "kube-api-access-6lnz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:16.337709 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.337642 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs" pod "511048c8-8540-4625-836a-dac96d03f292" (UID: "511048c8-8540-4625-836a-dac96d03f292"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:16.431428 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.431390 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p"] Nov 02 18:50:16.437416 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.437380 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511048c8-8540-4625-836a-dac96d03f292-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.437416 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.437406 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lnz6\" (UniqueName: \"kubernetes.io/projected/511048c8-8540-4625-836a-dac96d03f292-kube-api-access-6lnz6\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.437416 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.437416 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511048c8-8540-4625-836a-dac96d03f292-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:16.459199 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:50:16.459165 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a507175_a27c_406d_a42e_8bed1d13b219.slice/crio-8a53aa1bd24dda348b5a4515d9c6a317d8e6d747292753032cc2d339430b95bc WatchSource:0}: Error finding container 8a53aa1bd24dda348b5a4515d9c6a317d8e6d747292753032cc2d339430b95bc: Status 404 returned error can't find the container with id 8a53aa1bd24dda348b5a4515d9c6a317d8e6d747292753032cc2d339430b95bc Nov 02 18:50:16.721913 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.721885 5647 generic.go:356] "Generic (PLEG): container finished" podID="511048c8-8540-4625-836a-dac96d03f292" containerID="2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118" exitCode=0 Nov 02 18:50:16.722056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.721939 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" Nov 02 18:50:16.722056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.721968 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" event={"ID":"511048c8-8540-4625-836a-dac96d03f292","Type":"ContainerDied","Data":"2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118"} Nov 02 18:50:16.722056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.722012 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m" event={"ID":"511048c8-8540-4625-836a-dac96d03f292","Type":"ContainerDied","Data":"7f9ed19921407e9c614aaba7456579abda2e7fa9c83c05917262b1d959450a2e"} Nov 02 18:50:16.722056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.722035 5647 scope.go:122] "RemoveContainer" containerID="2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118" Nov 02 18:50:16.723562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.723528 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" Nov 02 18:50:16.723562 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.723541 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8" event={"ID":"45f752bb-62ca-48a6-95a0-0c871d543dc6","Type":"ContainerDied","Data":"fdc187a2ad15fe9fc853330a4f1ed9c4b55ed237e23fff114460d8e12861ce4c"} Nov 02 18:50:16.723772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.723589 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc187a2ad15fe9fc853330a4f1ed9c4b55ed237e23fff114460d8e12861ce4c" Nov 02 18:50:16.725427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.725406 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" event={"ID":"1a507175-a27c-406d-a42e-8bed1d13b219","Type":"ContainerStarted","Data":"8a53aa1bd24dda348b5a4515d9c6a317d8e6d747292753032cc2d339430b95bc"} Nov 02 18:50:16.732841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.732817 5647 scope.go:122] "RemoveContainer" containerID="2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118" Nov 02 18:50:16.733210 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:16.733153 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118\": container with ID starting with 2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118 not found: ID does not exist" containerID="2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118" Nov 02 18:50:16.733210 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.733183 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118"} err="failed to get container status \"2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118\": rpc error: code = NotFound desc = could not find container \"2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118\": container with ID starting with 2277737719eb30224aab573b3c99588cab39b5af203eae0d6c6df81dae004118 not found: ID does not exist" Nov 02 18:50:16.890139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.890030 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m"] Nov 02 18:50:16.894631 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:16.894406 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ceb1ae/premium-w0-ceb1ae-kserve-6874d5bccf-dfp4m"] Nov 02 18:50:17.008648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.008531 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" podStartSLOduration=2.008508782 podStartE2EDuration="2.008508782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:50:17.002278234 +0000 UTC m=+5204.075805451" watchObservedRunningTime="2026-11-02 18:50:17.008508782 +0000 UTC m=+5204.082036013" Nov 02 18:50:17.059615 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.059572 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2"] Nov 02 18:50:17.060663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.060631 5647 cpu_manager.go:414] "containerMap: removing container" podUID="45f752bb-62ca-48a6-95a0-0c871d543dc6" containerName="revoke-keys" Nov 02 18:50:17.060663 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.060661 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f752bb-62ca-48a6-95a0-0c871d543dc6" containerName="revoke-keys" Nov 02 18:50:17.060936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.060714 5647 cpu_manager.go:414] "containerMap: removing container" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" Nov 02 18:50:17.060936 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.060727 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" Nov 02 18:50:17.061037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.060985 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f752bb-62ca-48a6-95a0-0c871d543dc6" containerName="revoke-keys" Nov 02 18:50:17.061037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.061012 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="511048c8-8540-4625-836a-dac96d03f292" containerName="main" Nov 02 18:50:17.066720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.066681 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.082335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.082306 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="511048c8-8540-4625-836a-dac96d03f292" path="/var/lib/kubelet/pods/511048c8-8540-4625-836a-dac96d03f292/volumes" Nov 02 18:50:17.117002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.116972 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2"] Nov 02 18:50:17.145714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.145635 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.145714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.145679 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.146012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.145910 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzkk\" (UniqueName: \"kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.246895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.246860 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzkk\" (UniqueName: \"kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.247116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.246986 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.247116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.247036 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.247644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.247620 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.249864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.249837 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.311824 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.311577 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzkk\" (UniqueName: \"kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk\") pod \"maas-api-revoke-sub-a38e46a3-z6mf2\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.392519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.392274 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:17.403408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.403111 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:50:17.458694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.456112 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc"] Nov 02 18:50:17.464776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.464724 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m"] Nov 02 18:50:17.552817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.552779 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.552980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.552825 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.552980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.552865 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.552980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.552895 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.552980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.552920 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.553153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.552980 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.553153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553028 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.553153 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553093 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.553296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553156 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy\") pod \"0324d641-3db4-4347-93ff-f68b9fb6f913\" (UID: \"0324d641-3db4-4347-93ff-f68b9fb6f913\") " Nov 02 18:50:17.553352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553316 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:17.553809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553545 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:17.553809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553682 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.553809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553708 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0324d641-3db4-4347-93ff-f68b9fb6f913-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.553809 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553773 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:17.554309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553872 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:17.554309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.553880 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:17.555857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.555824 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:50:17.556224 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.556180 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:17.557164 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.557127 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "kube-api-access-q7nph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:17.557278 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.557129 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token" pod "0324d641-3db4-4347-93ff-f68b9fb6f913" (UID: "0324d641-3db4-4347-93ff-f68b9fb6f913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654863 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654894 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654905 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7nph\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-kube-api-access-q7nph\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654916 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654925 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654933 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.654956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.654941 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0324d641-3db4-4347-93ff-f68b9fb6f913-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:17.733302 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.733264 5647 generic.go:356] "Generic (PLEG): container finished" podID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerID="04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e" exitCode=0 Nov 02 18:50:17.733867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.733322 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" Nov 02 18:50:17.733867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.733361 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" event={"ID":"0324d641-3db4-4347-93ff-f68b9fb6f913","Type":"ContainerDied","Data":"04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e"} Nov 02 18:50:17.733867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.733422 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9" event={"ID":"0324d641-3db4-4347-93ff-f68b9fb6f913","Type":"ContainerDied","Data":"c6093238705e069322c84a563b96d4fb23349dfc994ac236b6ac3ee80ec7819b"} Nov 02 18:50:17.733867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.733444 5647 scope.go:122] "RemoveContainer" containerID="04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e" Nov 02 18:50:17.735307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.735283 5647 generic.go:356] "Generic (PLEG): container finished" podID="1a507175-a27c-406d-a42e-8bed1d13b219" containerID="d596dec56fe761047ee8219538d96e309ffd052898c72804c6bb95549bedc444" exitCode=0 Nov 02 18:50:17.735431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.735339 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" event={"ID":"1a507175-a27c-406d-a42e-8bed1d13b219","Type":"ContainerDied","Data":"d596dec56fe761047ee8219538d96e309ffd052898c72804c6bb95549bedc444"} Nov 02 18:50:17.744195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.744175 5647 scope.go:122] "RemoveContainer" containerID="4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a" Nov 02 18:50:17.754899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.754870 5647 scope.go:122] "RemoveContainer" containerID="04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e" Nov 02 18:50:17.755324 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:17.755290 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e\": container with ID starting with 04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e not found: ID does not exist" containerID="04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e" Nov 02 18:50:17.755432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.755333 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e"} err="failed to get container status \"04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e\": rpc error: code = NotFound desc = could not find container \"04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e\": container with ID starting with 04dd9a4b957ab4828a93d98e4fb43961bd4691d8fa4cd27e1f7caa8a6ad6f48e not found: ID does not exist" Nov 02 18:50:17.755432 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.755352 5647 scope.go:122] "RemoveContainer" containerID="4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a" Nov 02 18:50:17.755779 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:17.755719 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a\": container with ID starting with 4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a not found: ID does not exist" containerID="4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a" Nov 02 18:50:17.755867 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.755791 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a"} err="failed to get container status \"4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a\": rpc error: code = NotFound desc = could not find container \"4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a\": container with ID starting with 4def23f86eb58c6f650f21a56a59ac61fb2f8747c1f5565546f46d9f1b6b052a not found: ID does not exist" Nov 02 18:50:17.797720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.797695 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2"] Nov 02 18:50:17.798610 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:50:17.798584 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd449d2_8165_44d5_b1b6_28d774308dc9.slice/crio-c02eb628f591958129e1d4c266eb913252ca12e04a75ac19d20fd2b0f492f48f WatchSource:0}: Error finding container c02eb628f591958129e1d4c266eb913252ca12e04a75ac19d20fd2b0f492f48f: Status 404 returned error can't find the container with id c02eb628f591958129e1d4c266eb913252ca12e04a75ac19d20fd2b0f492f48f Nov 02 18:50:17.873553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.873519 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9"] Nov 02 18:50:17.877646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:17.877621 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c82472-openshift-default-56fbb9fb5c-nfcl9"] Nov 02 18:50:18.742433 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.742395 5647 generic.go:356] "Generic (PLEG): container finished" podID="7cd449d2-8165-44d5-b1b6-28d774308dc9" containerID="7347a37de8774ed59e742ffb8e4b51a96058fcccb525e729903ccb20cd1ad452" exitCode=0 Nov 02 18:50:18.742912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.742484 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" event={"ID":"7cd449d2-8165-44d5-b1b6-28d774308dc9","Type":"ContainerDied","Data":"7347a37de8774ed59e742ffb8e4b51a96058fcccb525e729903ccb20cd1ad452"} Nov 02 18:50:18.742912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.742550 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" event={"ID":"7cd449d2-8165-44d5-b1b6-28d774308dc9","Type":"ContainerStarted","Data":"c02eb628f591958129e1d4c266eb913252ca12e04a75ac19d20fd2b0f492f48f"} Nov 02 18:50:18.880339 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.880310 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:18.966760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.966642 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89\" (UniqueName: \"kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89\") pod \"1a507175-a27c-406d-a42e-8bed1d13b219\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " Nov 02 18:50:18.966760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.966706 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca\") pod \"1a507175-a27c-406d-a42e-8bed1d13b219\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " Nov 02 18:50:18.966964 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.966875 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca\") pod \"1a507175-a27c-406d-a42e-8bed1d13b219\" (UID: \"1a507175-a27c-406d-a42e-8bed1d13b219\") " Nov 02 18:50:18.967281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.967245 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca" pod "1a507175-a27c-406d-a42e-8bed1d13b219" (UID: "1a507175-a27c-406d-a42e-8bed1d13b219"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:18.969037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.969009 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89" pod "1a507175-a27c-406d-a42e-8bed1d13b219" (UID: "1a507175-a27c-406d-a42e-8bed1d13b219"). InnerVolumeSpecName "kube-api-access-lgz89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:18.969037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:18.969018 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca" pod "1a507175-a27c-406d-a42e-8bed1d13b219" (UID: "1a507175-a27c-406d-a42e-8bed1d13b219"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:19.068026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.067993 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:19.068026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.068023 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgz89\" (UniqueName: \"kubernetes.io/projected/1a507175-a27c-406d-a42e-8bed1d13b219-kube-api-access-lgz89\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:19.068026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.068034 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a507175-a27c-406d-a42e-8bed1d13b219-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:19.081194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.081165 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" path="/var/lib/kubelet/pods/0324d641-3db4-4347-93ff-f68b9fb6f913/volumes" Nov 02 18:50:19.750947 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.750915 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" Nov 02 18:50:19.751457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.750949 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p" event={"ID":"1a507175-a27c-406d-a42e-8bed1d13b219","Type":"ContainerDied","Data":"8a53aa1bd24dda348b5a4515d9c6a317d8e6d747292753032cc2d339430b95bc"} Nov 02 18:50:19.751457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.750999 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a53aa1bd24dda348b5a4515d9c6a317d8e6d747292753032cc2d339430b95bc" Nov 02 18:50:19.896301 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.896269 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:19.976906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.976869 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca\") pod \"7cd449d2-8165-44d5-b1b6-28d774308dc9\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " Nov 02 18:50:19.977100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.976957 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca\") pod \"7cd449d2-8165-44d5-b1b6-28d774308dc9\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " Nov 02 18:50:19.977100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.977002 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk\" (UniqueName: \"kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk\") pod \"7cd449d2-8165-44d5-b1b6-28d774308dc9\" (UID: \"7cd449d2-8165-44d5-b1b6-28d774308dc9\") " Nov 02 18:50:19.977427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.977392 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca" pod "7cd449d2-8165-44d5-b1b6-28d774308dc9" (UID: "7cd449d2-8165-44d5-b1b6-28d774308dc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:19.979175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.979146 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca" pod "7cd449d2-8165-44d5-b1b6-28d774308dc9" (UID: "7cd449d2-8165-44d5-b1b6-28d774308dc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:19.979318 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:19.979187 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk" pod "7cd449d2-8165-44d5-b1b6-28d774308dc9" (UID: "7cd449d2-8165-44d5-b1b6-28d774308dc9"). InnerVolumeSpecName "kube-api-access-rfzkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:20.078431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:20.078335 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:20.078431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:20.078368 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd449d2-8165-44d5-b1b6-28d774308dc9-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:20.078431 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:20.078383 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfzkk\" (UniqueName: \"kubernetes.io/projected/7cd449d2-8165-44d5-b1b6-28d774308dc9-kube-api-access-rfzkk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:20.758935 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:20.758901 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" Nov 02 18:50:20.759398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:20.758944 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2" event={"ID":"7cd449d2-8165-44d5-b1b6-28d774308dc9","Type":"ContainerDied","Data":"c02eb628f591958129e1d4c266eb913252ca12e04a75ac19d20fd2b0f492f48f"} Nov 02 18:50:20.759398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:20.759001 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c02eb628f591958129e1d4c266eb913252ca12e04a75ac19d20fd2b0f492f48f" Nov 02 18:50:23.842639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:23.842600 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:50:26.047375 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:26.047337 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv"] Nov 02 18:50:26.098094 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:26.098043 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dec28c4f-rmzfv"] Nov 02 18:50:27.079505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:27.079465 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c41009-5028-4ea6-bd5d-85fbcb95cbe1" path="/var/lib/kubelet/pods/34c41009-5028-4ea6-bd5d-85fbcb95cbe1/volumes" Nov 02 18:50:28.956088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:28.956048 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp"] Nov 02 18:50:28.956541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:28.956285 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" containerID="cri-o://78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e" gracePeriod=30 Nov 02 18:50:29.898616 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.898569 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh"] Nov 02 18:50:29.899047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899029 5647 cpu_manager.go:414] "containerMap: removing container" podUID="7cd449d2-8165-44d5-b1b6-28d774308dc9" containerName="revoke-keys" Nov 02 18:50:29.899047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899046 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd449d2-8165-44d5-b1b6-28d774308dc9" containerName="revoke-keys" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899067 5647 cpu_manager.go:414] "containerMap: removing container" podUID="1a507175-a27c-406d-a42e-8bed1d13b219" containerName="revoke-keys" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899073 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a507175-a27c-406d-a42e-8bed1d13b219" containerName="revoke-keys" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899081 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899085 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899101 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899105 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899170 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd449d2-8165-44d5-b1b6-28d774308dc9" containerName="revoke-keys" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899182 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a507175-a27c-406d-a42e-8bed1d13b219" containerName="revoke-keys" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899187 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" Nov 02 18:50:29.899213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.899195 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0324d641-3db4-4347-93ff-f68b9fb6f913" containerName="istio-proxy" Nov 02 18:50:29.903721 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.903700 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:29.959445 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:29.959412 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh"] Nov 02 18:50:30.007675 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.007639 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:50:30.008274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.008244 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:50:30.085029 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.084996 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szf9z\" (UniqueName: \"kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.085219 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.085052 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.085219 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.085159 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.186326 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.186229 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szf9z\" (UniqueName: \"kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.186507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.186353 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.186507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.186443 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.187237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.187209 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.189271 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.189248 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.243650 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.243623 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szf9z\" (UniqueName: \"kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z\") pod \"maas-api-revoke-sub-f198960e-6twjh\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.517878 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.517790 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:30.895166 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:30.895141 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh"] Nov 02 18:50:30.896052 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:50:30.896029 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937f82a5_5e88_4b89_a712_b11a67803cbc.slice/crio-d671298cdabb49a134266b4e65b1ee8ab5e40abaeb5ed3139fc4e30a85a2e3b1 WatchSource:0}: Error finding container d671298cdabb49a134266b4e65b1ee8ab5e40abaeb5ed3139fc4e30a85a2e3b1: Status 404 returned error can't find the container with id d671298cdabb49a134266b4e65b1ee8ab5e40abaeb5ed3139fc4e30a85a2e3b1 Nov 02 18:50:31.833886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:31.833850 5647 generic.go:356] "Generic (PLEG): container finished" podID="937f82a5-5e88-4b89-a712-b11a67803cbc" containerID="5454bbeb9b03a7104142cb4e8c523fb38214634cf4dc46e944ebb77778913e08" exitCode=0 Nov 02 18:50:31.834321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:31.833902 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" event={"ID":"937f82a5-5e88-4b89-a712-b11a67803cbc","Type":"ContainerDied","Data":"5454bbeb9b03a7104142cb4e8c523fb38214634cf4dc46e944ebb77778913e08"} Nov 02 18:50:31.834321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:31.833940 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" event={"ID":"937f82a5-5e88-4b89-a712-b11a67803cbc","Type":"ContainerStarted","Data":"d671298cdabb49a134266b4e65b1ee8ab5e40abaeb5ed3139fc4e30a85a2e3b1"} Nov 02 18:50:32.486372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.486326 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" containerID="cri-o://35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7" gracePeriod=45 Nov 02 18:50:32.502718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.502675 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" containerID="cri-o://89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af" gracePeriod=45 Nov 02 18:50:32.781108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.781082 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 18:50:32.807246 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.807223 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 18:50:32.840797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.840757 5647 generic.go:356] "Generic (PLEG): container finished" podID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerID="35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7" exitCode=0 Nov 02 18:50:32.841288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.840818 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" Nov 02 18:50:32.841288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.840848 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" event={"ID":"0f5bbe3c-b8e2-47b4-bab9-651eecceed79","Type":"ContainerDied","Data":"35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7"} Nov 02 18:50:32.841288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.840893 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc" event={"ID":"0f5bbe3c-b8e2-47b4-bab9-651eecceed79","Type":"ContainerDied","Data":"fe5d6a8c42354fb8f9e1af4c8d77f58c10cacb4e45d4dcff66295afd61880be5"} Nov 02 18:50:32.841288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.840909 5647 scope.go:122] "RemoveContainer" containerID="35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7" Nov 02 18:50:32.842502 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.842476 5647 generic.go:356] "Generic (PLEG): container finished" podID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerID="89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af" exitCode=0 Nov 02 18:50:32.842641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.842521 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" Nov 02 18:50:32.842641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.842565 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" event={"ID":"0a942f59-c314-41d8-aa5f-d46eae01060e","Type":"ContainerDied","Data":"89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af"} Nov 02 18:50:32.842641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.842605 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m" event={"ID":"0a942f59-c314-41d8-aa5f-d46eae01060e","Type":"ContainerDied","Data":"a792bf0270100b28f3748315f4f3468f3327b2175c4596614ae8167014b1105c"} Nov 02 18:50:32.852528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.852506 5647 scope.go:122] "RemoveContainer" containerID="35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7" Nov 02 18:50:32.852938 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:32.852915 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7\": container with ID starting with 35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7 not found: ID does not exist" containerID="35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7" Nov 02 18:50:32.853007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.852951 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7"} err="failed to get container status \"35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7\": rpc error: code = NotFound desc = could not find container \"35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7\": container with ID starting with 35838ac9ceab760ad86c6c9af53eb9a0b4e9f4a92e86c0d393a35dfce3bae2c7 not found: ID does not exist" Nov 02 18:50:32.853007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.852974 5647 scope.go:122] "RemoveContainer" containerID="89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af" Nov 02 18:50:32.862657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.862626 5647 scope.go:122] "RemoveContainer" containerID="89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af" Nov 02 18:50:32.863077 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:32.863050 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af\": container with ID starting with 89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af not found: ID does not exist" containerID="89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af" Nov 02 18:50:32.863185 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.863086 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af"} err="failed to get container status \"89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af\": rpc error: code = NotFound desc = could not find container \"89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af\": container with ID starting with 89b5da09c01e31eda681fd17662b7e0f2a8ea152745f5d0c26b27be8c075a7af not found: ID does not exist" Nov 02 18:50:32.911247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911220 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir\") pod \"0a942f59-c314-41d8-aa5f-d46eae01060e\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911269 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9\" (UniqueName: \"kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9\") pod \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911311 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home\") pod \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911355 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache\") pod \"0a942f59-c314-41d8-aa5f-d46eae01060e\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911401 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm\") pod \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911498 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs\") pod \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911542 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir\") pod \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911576 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm\") pod \"0a942f59-c314-41d8-aa5f-d46eae01060e\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911601 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs\") pod \"0a942f59-c314-41d8-aa5f-d46eae01060e\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911656 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache\") pod \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\" (UID: \"0f5bbe3c-b8e2-47b4-bab9-651eecceed79\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911681 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home\") pod \"0a942f59-c314-41d8-aa5f-d46eae01060e\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.911710 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b\" (UniqueName: \"kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b\") pod \"0a942f59-c314-41d8-aa5f-d46eae01060e\" (UID: \"0a942f59-c314-41d8-aa5f-d46eae01060e\") " Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.912881 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir" pod "0f5bbe3c-b8e2-47b4-bab9-651eecceed79" (UID: "0f5bbe3c-b8e2-47b4-bab9-651eecceed79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.913158 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home" pod "0a942f59-c314-41d8-aa5f-d46eae01060e" (UID: "0a942f59-c314-41d8-aa5f-d46eae01060e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.913321 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir" pod "0a942f59-c314-41d8-aa5f-d46eae01060e" (UID: "0a942f59-c314-41d8-aa5f-d46eae01060e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.913500 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home" pod "0f5bbe3c-b8e2-47b4-bab9-651eecceed79" (UID: "0f5bbe3c-b8e2-47b4-bab9-651eecceed79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.913527 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache" pod "0a942f59-c314-41d8-aa5f-d46eae01060e" (UID: "0a942f59-c314-41d8-aa5f-d46eae01060e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.914905 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.913859 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache" pod "0f5bbe3c-b8e2-47b4-bab9-651eecceed79" (UID: "0f5bbe3c-b8e2-47b4-bab9-651eecceed79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.916363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.916100 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b" pod "0a942f59-c314-41d8-aa5f-d46eae01060e" (UID: "0a942f59-c314-41d8-aa5f-d46eae01060e"). InnerVolumeSpecName "kube-api-access-rzw5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:32.917800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.917774 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs" pod "0a942f59-c314-41d8-aa5f-d46eae01060e" (UID: "0a942f59-c314-41d8-aa5f-d46eae01060e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:32.918296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.918276 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm" pod "0f5bbe3c-b8e2-47b4-bab9-651eecceed79" (UID: "0f5bbe3c-b8e2-47b4-bab9-651eecceed79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.918748 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.918715 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs" pod "0f5bbe3c-b8e2-47b4-bab9-651eecceed79" (UID: "0f5bbe3c-b8e2-47b4-bab9-651eecceed79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:32.919637 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.919614 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9" pod "0f5bbe3c-b8e2-47b4-bab9-651eecceed79" (UID: "0f5bbe3c-b8e2-47b4-bab9-651eecceed79"). InnerVolumeSpecName "kube-api-access-v8qr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:32.919637 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.919621 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm" pod "0a942f59-c314-41d8-aa5f-d46eae01060e" (UID: "0a942f59-c314-41d8-aa5f-d46eae01060e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:32.967149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:32.967117 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012780 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012810 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012822 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzw5b\" (UniqueName: \"kubernetes.io/projected/0a942f59-c314-41d8-aa5f-d46eae01060e-kube-api-access-rzw5b\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012830 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012840 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8qr9\" (UniqueName: \"kubernetes.io/projected/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-kube-api-access-v8qr9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012848 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012856 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012863 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.012865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012871 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.013294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012878 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5bbe3c-b8e2-47b4-bab9-651eecceed79-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.013294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012886 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a942f59-c314-41d8-aa5f-d46eae01060e-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.013294 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.012893 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a942f59-c314-41d8-aa5f-d46eae01060e-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.113643 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.113615 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z\" (UniqueName: \"kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z\") pod \"937f82a5-5e88-4b89-a712-b11a67803cbc\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " Nov 02 18:50:33.113823 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.113695 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca\") pod \"937f82a5-5e88-4b89-a712-b11a67803cbc\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " Nov 02 18:50:33.113886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.113871 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca\") pod \"937f82a5-5e88-4b89-a712-b11a67803cbc\" (UID: \"937f82a5-5e88-4b89-a712-b11a67803cbc\") " Nov 02 18:50:33.114108 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.114079 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca" pod "937f82a5-5e88-4b89-a712-b11a67803cbc" (UID: "937f82a5-5e88-4b89-a712-b11a67803cbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:33.115941 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.115910 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z" pod "937f82a5-5e88-4b89-a712-b11a67803cbc" (UID: "937f82a5-5e88-4b89-a712-b11a67803cbc"). InnerVolumeSpecName "kube-api-access-szf9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:33.116055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.115949 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca" pod "937f82a5-5e88-4b89-a712-b11a67803cbc" (UID: "937f82a5-5e88-4b89-a712-b11a67803cbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:50:33.214792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.214754 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.214792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.214788 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/937f82a5-5e88-4b89-a712-b11a67803cbc-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.214792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.214799 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szf9z\" (UniqueName: \"kubernetes.io/projected/937f82a5-5e88-4b89-a712-b11a67803cbc-kube-api-access-szf9z\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:33.298303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.298226 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc"] Nov 02 18:50:33.303944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.303921 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-2-w2-a099c3-kserve-859f8899df-tmglc"] Nov 02 18:50:33.413563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.413525 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m"] Nov 02 18:50:33.420714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.420678 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a099c3/distinct-w2-a099c3-kserve-89c85d586-gkk8m"] Nov 02 18:50:33.851719 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.851689 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" Nov 02 18:50:33.852276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.851684 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh" event={"ID":"937f82a5-5e88-4b89-a712-b11a67803cbc","Type":"ContainerDied","Data":"d671298cdabb49a134266b4e65b1ee8ab5e40abaeb5ed3139fc4e30a85a2e3b1"} Nov 02 18:50:33.852276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:33.851856 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d671298cdabb49a134266b4e65b1ee8ab5e40abaeb5ed3139fc4e30a85a2e3b1" Nov 02 18:50:34.212103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.212079 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 18:50:34.325712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325677 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.325712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325719 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.325957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325774 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.325957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325815 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.325957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325850 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.325957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325907 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.325957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325952 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.326245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.325987 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.326245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326012 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs\") pod \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\" (UID: \"f8ac16ac-2d3e-4811-a96d-94a6750f77ef\") " Nov 02 18:50:34.326245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326045 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:34.326427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326275 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:34.326427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326305 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:50:34.326539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326412 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.326539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326451 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.326539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326466 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.326539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326495 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:34.326539 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.326482 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:34.328117 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.328090 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:50:34.328245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.328165 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:34.328625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.328605 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:50:34.328686 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.328619 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv" pod "f8ac16ac-2d3e-4811-a96d-94a6750f77ef" (UID: "f8ac16ac-2d3e-4811-a96d-94a6750f77ef"). InnerVolumeSpecName "kube-api-access-slzxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:50:34.427101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.427010 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.427101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.427045 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slzxv\" (UniqueName: \"kubernetes.io/projected/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-kube-api-access-slzxv\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.427101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.427058 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.427101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.427070 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.427101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.427079 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.427101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.427088 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ac16ac-2d3e-4811-a96d-94a6750f77ef-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:50:34.861534 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.861501 5647 generic.go:356] "Generic (PLEG): container finished" podID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerID="78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e" exitCode=0 Nov 02 18:50:34.861999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.861553 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" Nov 02 18:50:34.861999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.861593 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" event={"ID":"f8ac16ac-2d3e-4811-a96d-94a6750f77ef","Type":"ContainerDied","Data":"78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e"} Nov 02 18:50:34.861999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.861653 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp" event={"ID":"f8ac16ac-2d3e-4811-a96d-94a6750f77ef","Type":"ContainerDied","Data":"5e4070e2c6c4cd832d909ef0f67ce863c4e2b8add1e0a43ad79fa03a6b5f841f"} Nov 02 18:50:34.861999 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.861681 5647 scope.go:122] "RemoveContainer" containerID="78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e" Nov 02 18:50:34.872046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.872029 5647 scope.go:122] "RemoveContainer" containerID="09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506" Nov 02 18:50:34.880940 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.880923 5647 scope.go:122] "RemoveContainer" containerID="78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e" Nov 02 18:50:34.881235 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:34.881212 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e\": container with ID starting with 78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e not found: ID does not exist" containerID="78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e" Nov 02 18:50:34.881314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.881246 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e"} err="failed to get container status \"78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e\": rpc error: code = NotFound desc = could not find container \"78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e\": container with ID starting with 78e6de090d58592c74b6cf7163fb50481c183549d3061f6a8575809b0d50f49e not found: ID does not exist" Nov 02 18:50:34.881314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.881266 5647 scope.go:122] "RemoveContainer" containerID="09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506" Nov 02 18:50:34.881545 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:34.881525 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506\": container with ID starting with 09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506 not found: ID does not exist" containerID="09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506" Nov 02 18:50:34.881596 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:34.881552 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506"} err="failed to get container status \"09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506\": rpc error: code = NotFound desc = could not find container \"09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506\": container with ID starting with 09046a61b9831b43d23fbd960e549a1576c4c19b30795fb8f258282bd2364506 not found: ID does not exist" Nov 02 18:50:35.028572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:35.028535 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp"] Nov 02 18:50:35.082169 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:35.082136 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" path="/var/lib/kubelet/pods/0a942f59-c314-41d8-aa5f-d46eae01060e/volumes" Nov 02 18:50:35.082549 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:35.082536 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" path="/var/lib/kubelet/pods/0f5bbe3c-b8e2-47b4-bab9-651eecceed79/volumes" Nov 02 18:50:35.082865 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:35.082851 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dc5689-openshift-default-78ddb9bcc7-n4flp"] Nov 02 18:50:35.948644 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:50:35.948607 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:50:37.080293 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:37.080260 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" path="/var/lib/kubelet/pods/f8ac16ac-2d3e-4811-a96d-94a6750f77ef/volumes" Nov 02 18:50:39.045524 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:39.045487 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4"] Nov 02 18:50:39.070958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:39.070930 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c74266d4-dgws4"] Nov 02 18:50:39.079922 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:50:39.079895 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4a26bf4-531b-4a9f-a586-bb0e1f39a756" path="/var/lib/kubelet/pods/d4a26bf4-531b-4a9f-a586-bb0e1f39a756/volumes" Nov 02 18:51:05.947319 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:51:05.947233 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:51:28.794532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:28.794493 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:51:35.946855 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:51:35.946806 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:51:36.046036 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:36.046006 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7"] Nov 02 18:51:36.096625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:36.096588 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1cb6108-tvqs7"] Nov 02 18:51:37.080020 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:37.079987 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd86b039-4e46-411e-a0bb-d2edc87cd252" path="/var/lib/kubelet/pods/dd86b039-4e46-411e-a0bb-d2edc87cd252/volumes" Nov 02 18:51:37.847430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:37.847397 5647 scope.go:122] "RemoveContainer" containerID="bf10623808b00e64dc8dacc6855e0cd8ec0dfd7a3b6cd57357a4bcca583c0344" Nov 02 18:51:37.857038 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:37.857008 5647 scope.go:122] "RemoveContainer" containerID="be3fb045b122b95b78bfc7657eb7d6a987ba870a1ee2656ab891cf8577c70243" Nov 02 18:51:40.421948 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.421907 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm"] Nov 02 18:51:40.422386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.422135 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" podUID="93d41eb7-accd-4b43-a7c7-121e38a885d8" containerName="maas-api" containerID="cri-o://0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950" gracePeriod=30 Nov 02 18:51:40.693007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.692978 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:51:40.791585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.791544 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs\") pod \"93d41eb7-accd-4b43-a7c7-121e38a885d8\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " Nov 02 18:51:40.792095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.792070 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca\") pod \"93d41eb7-accd-4b43-a7c7-121e38a885d8\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " Nov 02 18:51:40.793368 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.793338 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj\" (UniqueName: \"kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj\") pod \"93d41eb7-accd-4b43-a7c7-121e38a885d8\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " Nov 02 18:51:40.793500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.793409 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\") pod \"93d41eb7-accd-4b43-a7c7-121e38a885d8\" (UID: \"93d41eb7-accd-4b43-a7c7-121e38a885d8\") " Nov 02 18:51:40.794248 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.794213 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca" pod "93d41eb7-accd-4b43-a7c7-121e38a885d8" (UID: "93d41eb7-accd-4b43-a7c7-121e38a885d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:51:40.796232 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.796201 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs" pod "93d41eb7-accd-4b43-a7c7-121e38a885d8" (UID: "93d41eb7-accd-4b43-a7c7-121e38a885d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:51:40.796354 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.796302 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls" pod "93d41eb7-accd-4b43-a7c7-121e38a885d8" (UID: "93d41eb7-accd-4b43-a7c7-121e38a885d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:51:40.796418 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.796402 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj" pod "93d41eb7-accd-4b43-a7c7-121e38a885d8" (UID: "93d41eb7-accd-4b43-a7c7-121e38a885d8"). InnerVolumeSpecName "kube-api-access-tssmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:40.894858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.894819 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:40.894858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.894852 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93d41eb7-accd-4b43-a7c7-121e38a885d8-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:40.894858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.894861 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tssmj\" (UniqueName: \"kubernetes.io/projected/93d41eb7-accd-4b43-a7c7-121e38a885d8-kube-api-access-tssmj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:40.895124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:40.894871 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93d41eb7-accd-4b43-a7c7-121e38a885d8-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:41.243482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.243449 5647 generic.go:356] "Generic (PLEG): container finished" podID="93d41eb7-accd-4b43-a7c7-121e38a885d8" containerID="0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950" exitCode=0 Nov 02 18:51:41.243673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.243507 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" Nov 02 18:51:41.243673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.243539 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" event={"ID":"93d41eb7-accd-4b43-a7c7-121e38a885d8","Type":"ContainerDied","Data":"0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950"} Nov 02 18:51:41.243673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.243596 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm" event={"ID":"93d41eb7-accd-4b43-a7c7-121e38a885d8","Type":"ContainerDied","Data":"c37b87f06efac1e1c4e840256f915703041fce6ae9d51baa72e440cc208efa6f"} Nov 02 18:51:41.243673 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.243619 5647 scope.go:122] "RemoveContainer" containerID="0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950" Nov 02 18:51:41.252980 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.252962 5647 scope.go:122] "RemoveContainer" containerID="0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950" Nov 02 18:51:41.253247 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:51:41.253226 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950\": container with ID starting with 0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950 not found: ID does not exist" containerID="0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950" Nov 02 18:51:41.253295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.253257 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950"} err="failed to get container status \"0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950\": rpc error: code = NotFound desc = could not find container \"0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950\": container with ID starting with 0f0c41ab993bec0ff57d244d4c4c70595b99dd0393c1807bfe193845b9180950 not found: ID does not exist" Nov 02 18:51:41.411230 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.411138 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm"] Nov 02 18:51:41.414145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:41.414121 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f7aa2e-68c99bfd86-rtntm"] Nov 02 18:51:42.044272 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:42.044230 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb"] Nov 02 18:51:42.046572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:42.046542 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86c6d29f-b8mnb"] Nov 02 18:51:43.080034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:43.080005 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d41eb7-accd-4b43-a7c7-121e38a885d8" path="/var/lib/kubelet/pods/93d41eb7-accd-4b43-a7c7-121e38a885d8/volumes" Nov 02 18:51:43.080440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:43.080363 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf51689-3cc6-4f12-afcf-044f0ba3fdb2" path="/var/lib/kubelet/pods/adf51689-3cc6-4f12-afcf-044f0ba3fdb2/volumes" Nov 02 18:51:46.045919 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:46.045888 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6"] Nov 02 18:51:46.081569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:46.081537 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88e19cbb-vjjm6"] Nov 02 18:51:46.151219 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:46.151176 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:51:47.080106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:47.080070 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e175876-c12c-433e-ad08-432325ac918c" path="/var/lib/kubelet/pods/6e175876-c12c-433e-ad08-432325ac918c/volumes" Nov 02 18:51:51.732090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:51.732059 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm"] Nov 02 18:51:51.732560 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:51.732266 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" containerID="cri-o://dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc" gracePeriod=30 Nov 02 18:51:53.045338 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:53.045299 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8"] Nov 02 18:51:53.095791 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:53.095758 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8"] Nov 02 18:51:53.128428 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:51:53.128393 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-d495f970-g8xg8\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="68586401-3cfb-4d7e-9a0b-22f2e45a100e" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d495f970-g8xg8" Nov 02 18:51:55.079710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:55.079675 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68586401-3cfb-4d7e-9a0b-22f2e45a100e" path="/var/lib/kubelet/pods/68586401-3cfb-4d7e-9a0b-22f2e45a100e/volumes" Nov 02 18:51:56.046763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:56.046713 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht"] Nov 02 18:51:56.097124 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:56.097092 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3b13b50-jjpht"] Nov 02 18:51:56.991926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:56.991900 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:51:57.079372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.079282 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e48443-53cc-49fd-bfa0-8b8fe22683ec" path="/var/lib/kubelet/pods/98e48443-53cc-49fd-bfa0-8b8fe22683ec/volumes" Nov 02 18:51:57.165621 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165591 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165680 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165746 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165779 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165810 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165830 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165860 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165902 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165934 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:57.166106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.165941 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket\") pod \"6e44a8e2-8ff6-466d-a172-dd06de373029\" (UID: \"6e44a8e2-8ff6-466d-a172-dd06de373029\") " Nov 02 18:51:57.166950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.166142 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:57.166950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.166407 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.166950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.166429 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.166950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.166425 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:57.166950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.166490 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:51:57.166950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.166635 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:57.168780 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.168721 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:51:57.168912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.168851 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "kube-api-access-k44hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:57.168912 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.168891 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:51:57.169070 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.169057 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo" pod "6e44a8e2-8ff6-466d-a172-dd06de373029" (UID: "6e44a8e2-8ff6-466d-a172-dd06de373029"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:51:57.267684 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267647 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.267684 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267677 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.267684 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267687 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.267973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267695 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e44a8e2-8ff6-466d-a172-dd06de373029-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.267973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267707 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44hn\" (UniqueName: \"kubernetes.io/projected/6e44a8e2-8ff6-466d-a172-dd06de373029-kube-api-access-k44hn\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.267973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267716 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e44a8e2-8ff6-466d-a172-dd06de373029-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.267973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.267725 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e44a8e2-8ff6-466d-a172-dd06de373029-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:51:57.335916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.335822 5647 generic.go:356] "Generic (PLEG): container finished" podID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerID="dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc" exitCode=0 Nov 02 18:51:57.335916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.335894 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" Nov 02 18:51:57.335916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.335898 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" event={"ID":"6e44a8e2-8ff6-466d-a172-dd06de373029","Type":"ContainerDied","Data":"dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc"} Nov 02 18:51:57.336151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.335974 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm" event={"ID":"6e44a8e2-8ff6-466d-a172-dd06de373029","Type":"ContainerDied","Data":"5f0e666d3723d01e23fe746e63c9dfbf0e1d1bc76ab58cd44574d66ed1186d15"} Nov 02 18:51:57.336151 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.336007 5647 scope.go:122] "RemoveContainer" containerID="dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc" Nov 02 18:51:57.347040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.347017 5647 scope.go:122] "RemoveContainer" containerID="92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1" Nov 02 18:51:57.356064 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.356041 5647 scope.go:122] "RemoveContainer" containerID="dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc" Nov 02 18:51:57.356333 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:51:57.356311 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc\": container with ID starting with dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc not found: ID does not exist" containerID="dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc" Nov 02 18:51:57.356384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.356344 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc"} err="failed to get container status \"dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc\": rpc error: code = NotFound desc = could not find container \"dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc\": container with ID starting with dfaed8c53fdd1388c712487afab85d601ed144dacfece5b1ca4fca6e6f9dc5dc not found: ID does not exist" Nov 02 18:51:57.356384 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.356364 5647 scope.go:122] "RemoveContainer" containerID="92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1" Nov 02 18:51:57.356640 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:51:57.356619 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1\": container with ID starting with 92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1 not found: ID does not exist" containerID="92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1" Nov 02 18:51:57.356695 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.356645 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1"} err="failed to get container status \"92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1\": rpc error: code = NotFound desc = could not find container \"92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1\": container with ID starting with 92a49a8f528ea8c1cc801932724d402eb12792a7c05e9516c942339859f134a1 not found: ID does not exist" Nov 02 18:51:57.503711 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.503673 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm"] Nov 02 18:51:57.506670 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:57.506644 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f7aa2e-openshift-default-c87d98477-lsvwm"] Nov 02 18:51:59.080674 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:51:59.080642 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" path="/var/lib/kubelet/pods/6e44a8e2-8ff6-466d-a172-dd06de373029/volumes" Nov 02 18:52:05.043499 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:05.043463 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w"] Nov 02 18:52:05.093958 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:05.093924 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee8b00e-2vm7w"] Nov 02 18:52:05.947187 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:52:05.947151 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:52:06.044921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:06.044887 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm"] Nov 02 18:52:06.095452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:06.095418 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ea8f49a-5xwvm"] Nov 02 18:52:07.079952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:07.079916 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29c8c83-a731-45f4-8a15-d5a4f758091a" path="/var/lib/kubelet/pods/b29c8c83-a731-45f4-8a15-d5a4f758091a/volumes" Nov 02 18:52:07.080394 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:07.080316 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2b5cd6-c2d7-48bb-9956-e264c8679d80" path="/var/lib/kubelet/pods/df2b5cd6-c2d7-48bb-9956-e264c8679d80/volumes" Nov 02 18:52:09.045861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:09.045824 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl"] Nov 02 18:52:09.096324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:09.096292 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-571e3943-s78fl"] Nov 02 18:52:11.081125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:11.081088 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4" path="/var/lib/kubelet/pods/7391edd6-f08f-4e0b-a147-cdc2ac6d0bf4/volumes" Nov 02 18:52:15.045101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:15.045067 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw"] Nov 02 18:52:15.095811 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:15.095776 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95cb332f-hfhcw"] Nov 02 18:52:16.132450 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:16.132417 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:52:17.079372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:17.079339 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87e0c87-72ea-4da0-ad0a-ac8a7cf66574" path="/var/lib/kubelet/pods/e87e0c87-72ea-4da0-ad0a-ac8a7cf66574/volumes" Nov 02 18:52:21.137558 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.137523 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c"] Nov 02 18:52:21.138702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138665 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" Nov 02 18:52:21.138702 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138695 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138725 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138750 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138763 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138769 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138786 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138793 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138817 5647 cpu_manager.go:414] "containerMap: removing container" podUID="937f82a5-5e88-4b89-a712-b11a67803cbc" containerName="revoke-keys" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138825 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937f82a5-5e88-4b89-a712-b11a67803cbc" containerName="revoke-keys" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138841 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138847 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138865 5647 cpu_manager.go:414] "containerMap: removing container" podUID="93d41eb7-accd-4b43-a7c7-121e38a885d8" containerName="maas-api" Nov 02 18:52:21.138945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138872 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d41eb7-accd-4b43-a7c7-121e38a885d8" containerName="maas-api" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138979 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d41eb7-accd-4b43-a7c7-121e38a885d8" containerName="maas-api" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.138993 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.139004 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.139017 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.139026 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ac16ac-2d3e-4811-a96d-94a6750f77ef" containerName="istio-proxy" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.139037 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5bbe3c-b8e2-47b4-bab9-651eecceed79" containerName="main" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.139047 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="937f82a5-5e88-4b89-a712-b11a67803cbc" containerName="revoke-keys" Nov 02 18:52:21.139576 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.139056 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a942f59-c314-41d8-aa5f-d46eae01060e" containerName="main" Nov 02 18:52:21.142247 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.142224 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.173577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.173547 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.173792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.173626 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.173792 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.173642 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47h8d\" (UniqueName: \"kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.195364 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.195331 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c"] Nov 02 18:52:21.195548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.195514 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:52:21.195627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.195608 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:52:21.274914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.274882 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.274914 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.274921 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47h8d\" (UniqueName: \"kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.275177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.274982 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.275538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.275515 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.277496 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.277469 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.333056 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.333022 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47h8d\" (UniqueName: \"kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d\") pod \"maas-api-revoke-sub-78dfeaf0-zht6c\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.455569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.455478 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:21.832013 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:21.831987 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c"] Nov 02 18:52:21.833811 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:52:21.833715 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b1ef68_41e2_42a9_b4ad_4d70feae21a7.slice/crio-f6b2c69d8c23414778dbdfb6dabd2908fb45d8a890030e42012430ff386ab1b9 WatchSource:0}: Error finding container f6b2c69d8c23414778dbdfb6dabd2908fb45d8a890030e42012430ff386ab1b9: Status 404 returned error can't find the container with id f6b2c69d8c23414778dbdfb6dabd2908fb45d8a890030e42012430ff386ab1b9 Nov 02 18:52:22.482974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:22.482876 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" event={"ID":"19b1ef68-41e2-42a9-b4ad-4d70feae21a7","Type":"ContainerStarted","Data":"aefb4b15b9bd0609485ca6dc896aa2e85b9286c478586eb8e829c84d845212a3"} Nov 02 18:52:22.482974 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:22.482937 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" event={"ID":"19b1ef68-41e2-42a9-b4ad-4d70feae21a7","Type":"ContainerStarted","Data":"f6b2c69d8c23414778dbdfb6dabd2908fb45d8a890030e42012430ff386ab1b9"} Nov 02 18:52:22.601177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:22.601114 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" podStartSLOduration=1.601100489 podStartE2EDuration="1.601100489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:52:22.599538777 +0000 UTC m=+5329.673065981" watchObservedRunningTime="2026-11-02 18:52:22.601100489 +0000 UTC m=+5329.674627686" Nov 02 18:52:23.490488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:23.490454 5647 generic.go:356] "Generic (PLEG): container finished" podID="19b1ef68-41e2-42a9-b4ad-4d70feae21a7" containerID="aefb4b15b9bd0609485ca6dc896aa2e85b9286c478586eb8e829c84d845212a3" exitCode=0 Nov 02 18:52:23.490956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:23.490538 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" event={"ID":"19b1ef68-41e2-42a9-b4ad-4d70feae21a7","Type":"ContainerDied","Data":"aefb4b15b9bd0609485ca6dc896aa2e85b9286c478586eb8e829c84d845212a3"} Nov 02 18:52:24.630770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.630721 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:24.707193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.707164 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca\") pod \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " Nov 02 18:52:24.707419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.707269 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca\") pod \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " Nov 02 18:52:24.707419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.707370 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d\" (UniqueName: \"kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d\") pod \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\" (UID: \"19b1ef68-41e2-42a9-b4ad-4d70feae21a7\") " Nov 02 18:52:24.707646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.707619 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca" pod "19b1ef68-41e2-42a9-b4ad-4d70feae21a7" (UID: "19b1ef68-41e2-42a9-b4ad-4d70feae21a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:52:24.709372 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.709344 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca" pod "19b1ef68-41e2-42a9-b4ad-4d70feae21a7" (UID: "19b1ef68-41e2-42a9-b4ad-4d70feae21a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:52:24.709492 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.709463 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d" pod "19b1ef68-41e2-42a9-b4ad-4d70feae21a7" (UID: "19b1ef68-41e2-42a9-b4ad-4d70feae21a7"). InnerVolumeSpecName "kube-api-access-47h8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:52:24.808540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.808461 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47h8d\" (UniqueName: \"kubernetes.io/projected/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-kube-api-access-47h8d\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:24.808540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.808490 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:24.808540 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:24.808500 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b1ef68-41e2-42a9-b4ad-4d70feae21a7-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:25.493718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:25.493624 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g"] Nov 02 18:52:25.504634 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:25.504606 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" Nov 02 18:52:25.505109 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:25.504601 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c" event={"ID":"19b1ef68-41e2-42a9-b4ad-4d70feae21a7","Type":"ContainerDied","Data":"f6b2c69d8c23414778dbdfb6dabd2908fb45d8a890030e42012430ff386ab1b9"} Nov 02 18:52:25.505237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:25.505135 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b2c69d8c23414778dbdfb6dabd2908fb45d8a890030e42012430ff386ab1b9" Nov 02 18:52:31.394194 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:31.394156 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:52:32.947357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.947318 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf"] Nov 02 18:52:32.947808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.947750 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" Nov 02 18:52:32.947808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.947764 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e44a8e2-8ff6-466d-a172-dd06de373029" containerName="istio-proxy" Nov 02 18:52:32.947808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.947780 5647 cpu_manager.go:414] "containerMap: removing container" podUID="19b1ef68-41e2-42a9-b4ad-4d70feae21a7" containerName="revoke-keys" Nov 02 18:52:32.947808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.947785 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b1ef68-41e2-42a9-b4ad-4d70feae21a7" containerName="revoke-keys" Nov 02 18:52:32.947968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.947877 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b1ef68-41e2-42a9-b4ad-4d70feae21a7" containerName="revoke-keys" Nov 02 18:52:32.951204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:32.951176 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.005273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.005138 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:52:33.007058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.007031 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf"] Nov 02 18:52:33.055532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.055502 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:52:33.091424 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.091393 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.091599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.091435 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.091599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.091457 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gk9j\" (UniqueName: \"kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.192148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.192106 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.192360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.192266 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.192360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.192305 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gk9j\" (UniqueName: \"kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.192887 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.192870 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.244059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.243971 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:52:33.250427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.250406 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gk9j\" (UniqueName: \"kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.255195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.255169 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-048cb7fb-s2zbf\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.315217 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.315190 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq8lv\"" Nov 02 18:52:33.323996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.323974 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:33.695555 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:33.695525 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf"] Nov 02 18:52:33.698812 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:52:33.698769 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893f03c7_a0b1_4e81_a3a8_9d6fdc5c8b0d.slice/crio-c5130fc3a4c69b742424e3fb95a988db761e3d8a8e2d22e38c8c75cdf0c66416 WatchSource:0}: Error finding container c5130fc3a4c69b742424e3fb95a988db761e3d8a8e2d22e38c8c75cdf0c66416: Status 404 returned error can't find the container with id c5130fc3a4c69b742424e3fb95a988db761e3d8a8e2d22e38c8c75cdf0c66416 Nov 02 18:52:34.557930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:34.557882 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" event={"ID":"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d","Type":"ContainerStarted","Data":"e5ce56078d502f8bc9536073e8477ce0c17fcffb13752994100f2c7dedb28d21"} Nov 02 18:52:34.558464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:34.557962 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" event={"ID":"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d","Type":"ContainerStarted","Data":"c5130fc3a4c69b742424e3fb95a988db761e3d8a8e2d22e38c8c75cdf0c66416"} Nov 02 18:52:34.673812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:34.673756 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" podStartSLOduration=2.673741822 podStartE2EDuration="2.673741822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:52:34.672036742 +0000 UTC m=+5341.745563949" watchObservedRunningTime="2026-11-02 18:52:34.673741822 +0000 UTC m=+5341.747269018" Nov 02 18:52:35.571092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:35.571043 5647 generic.go:356] "Generic (PLEG): container finished" podID="893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" containerID="e5ce56078d502f8bc9536073e8477ce0c17fcffb13752994100f2c7dedb28d21" exitCode=0 Nov 02 18:52:35.571528 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:35.571161 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" event={"ID":"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d","Type":"ContainerDied","Data":"e5ce56078d502f8bc9536073e8477ce0c17fcffb13752994100f2c7dedb28d21"} Nov 02 18:52:35.946547 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:52:35.946510 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:52:36.713007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.712976 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:36.827816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.827713 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca\") pod \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " Nov 02 18:52:36.827816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.827774 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca\") pod \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " Nov 02 18:52:36.828067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.827889 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j\" (UniqueName: \"kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j\") pod \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\" (UID: \"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d\") " Nov 02 18:52:36.828381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.828351 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca" pod "893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" (UID: "893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:52:36.830330 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.830301 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca" pod "893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" (UID: "893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:52:36.831040 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.831015 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j" pod "893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" (UID: "893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d"). InnerVolumeSpecName "kube-api-access-8gk9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:52:36.929570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.929528 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gk9j\" (UniqueName: \"kubernetes.io/projected/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-kube-api-access-8gk9j\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:36.929570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.929563 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:36.929570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:36.929573 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:37.584296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:37.584207 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" Nov 02 18:52:37.584296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:37.584237 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf" event={"ID":"893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d","Type":"ContainerDied","Data":"c5130fc3a4c69b742424e3fb95a988db761e3d8a8e2d22e38c8c75cdf0c66416"} Nov 02 18:52:37.584296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:37.584293 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5130fc3a4c69b742424e3fb95a988db761e3d8a8e2d22e38c8c75cdf0c66416" Nov 02 18:52:37.882570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:37.882536 5647 scope.go:122] "RemoveContainer" containerID="7756afa58e6f0deab7b2f8d8f8ef87b962e3edce7b85ba02e64b05f93b2ae215" Nov 02 18:52:40.515774 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.515707 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" containerID="cri-o://56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb" gracePeriod=45 Nov 02 18:52:40.780365 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.780304 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 18:52:40.866221 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866186 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir\") pod \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " Nov 02 18:52:40.866404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866236 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm\") pod \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " Nov 02 18:52:40.866404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866272 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache\") pod \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " Nov 02 18:52:40.866404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866301 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs\") pod \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " Nov 02 18:52:40.866404 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866328 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home\") pod \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " Nov 02 18:52:40.866646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866424 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll\" (UniqueName: \"kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll\") pod \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\" (UID: \"20469e2c-c670-4c90-9102-6b4e3ecc8b7a\") " Nov 02 18:52:40.866646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866540 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache" pod "20469e2c-c670-4c90-9102-6b4e3ecc8b7a" (UID: "20469e2c-c670-4c90-9102-6b4e3ecc8b7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:52:40.866646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866532 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir" pod "20469e2c-c670-4c90-9102-6b4e3ecc8b7a" (UID: "20469e2c-c670-4c90-9102-6b4e3ecc8b7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:52:40.866854 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866699 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home" pod "20469e2c-c670-4c90-9102-6b4e3ecc8b7a" (UID: "20469e2c-c670-4c90-9102-6b4e3ecc8b7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:52:40.866917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866891 5647 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tmp-dir\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:40.866965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866913 5647 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-model-cache\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:40.866965 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.866927 5647 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-home\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:40.868855 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.868824 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs" pod "20469e2c-c670-4c90-9102-6b4e3ecc8b7a" (UID: "20469e2c-c670-4c90-9102-6b4e3ecc8b7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:52:40.869037 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.868925 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm" pod "20469e2c-c670-4c90-9102-6b4e3ecc8b7a" (UID: "20469e2c-c670-4c90-9102-6b4e3ecc8b7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:52:40.869871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.869843 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll" pod "20469e2c-c670-4c90-9102-6b4e3ecc8b7a" (UID: "20469e2c-c670-4c90-9102-6b4e3ecc8b7a"). InnerVolumeSpecName "kube-api-access-lbpll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:52:40.968381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.968331 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpll\" (UniqueName: \"kubernetes.io/projected/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-kube-api-access-lbpll\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:40.968381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.968369 5647 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-dshm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:40.968381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:40.968384 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20469e2c-c670-4c90-9102-6b4e3ecc8b7a-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:52:41.610168 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.610134 5647 generic.go:356] "Generic (PLEG): container finished" podID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerID="56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb" exitCode=0 Nov 02 18:52:41.610610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.610180 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" Nov 02 18:52:41.610610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.610221 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" event={"ID":"20469e2c-c670-4c90-9102-6b4e3ecc8b7a","Type":"ContainerDied","Data":"56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb"} Nov 02 18:52:41.610610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.610267 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g" event={"ID":"20469e2c-c670-4c90-9102-6b4e3ecc8b7a","Type":"ContainerDied","Data":"51218e383a3fd273dbb2fd478baef98a44eae4abb1eebc054d7b577233ed01d2"} Nov 02 18:52:41.610610 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.610289 5647 scope.go:122] "RemoveContainer" containerID="56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb" Nov 02 18:52:41.627618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.627596 5647 scope.go:122] "RemoveContainer" containerID="56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb" Nov 02 18:52:41.627952 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:52:41.627928 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb\": container with ID starting with 56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb not found: ID does not exist" containerID="56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb" Nov 02 18:52:41.628021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.627961 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb"} err="failed to get container status \"56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb\": rpc error: code = NotFound desc = could not find container \"56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb\": container with ID starting with 56cc345c4d836f9c158df9c11c7f3111071a3470bfec1aa41f169c809faf8afb not found: ID does not exist" Nov 02 18:52:41.783516 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.783485 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g"] Nov 02 18:52:41.833875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:41.833839 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-6e3175/distinct-2-w3-6e3175-kserve-7dfc447bd4-78c2g"] Nov 02 18:52:43.079328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:43.079293 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" path="/var/lib/kubelet/pods/20469e2c-c670-4c90-9102-6b4e3ecc8b7a/volumes" Nov 02 18:52:49.464016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:49.463967 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:52:50.337930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.337891 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb"] Nov 02 18:52:50.338296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.338283 5647 cpu_manager.go:414] "containerMap: removing container" podUID="893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" containerName="revoke-keys" Nov 02 18:52:50.338344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.338297 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" containerName="revoke-keys" Nov 02 18:52:50.338344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.338332 5647 cpu_manager.go:414] "containerMap: removing container" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" Nov 02 18:52:50.338344 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.338337 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" Nov 02 18:52:50.338453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.338399 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" containerName="revoke-keys" Nov 02 18:52:50.338453 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.338416 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="20469e2c-c670-4c90-9102-6b4e3ecc8b7a" containerName="main" Nov 02 18:52:50.341617 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.341596 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354019 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354067 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354136 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354169 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354212 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354189 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cptj\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354262 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354306 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354363 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.354437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.354400 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.392875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.392832 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb"] Nov 02 18:52:50.441139 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.441100 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-6e3175-openshift-default-dockercfg-n7khr\"" Nov 02 18:52:50.455328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455301 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455368 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455388 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455408 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455457 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455429 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455453 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cptj\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455580 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455631 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455660 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455830 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.455895 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455860 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.456019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455911 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.456019 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.455954 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.456308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.456285 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.458861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.458839 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.459002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.458835 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.513379 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.513343 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.513760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.513394 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cptj\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj\") pod \"e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:50.655227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:50.655131 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:51.083701 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.083678 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb"] Nov 02 18:52:51.085249 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:52:51.085211 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce9c5cf_236f_4d9c_86ca_ad0317f6a331.slice/crio-6ac2252bfcc5def4e720d81ebcfd0dc8d8615d491ab7e9118b16999d2ddc0517 WatchSource:0}: Error finding container 6ac2252bfcc5def4e720d81ebcfd0dc8d8615d491ab7e9118b16999d2ddc0517: Status 404 returned error can't find the container with id 6ac2252bfcc5def4e720d81ebcfd0dc8d8615d491ab7e9118b16999d2ddc0517 Nov 02 18:52:51.087319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.087286 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:52:51.087400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.087371 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:52:51.087448 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.087402 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:52:51.668807 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.668767 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" event={"ID":"cce9c5cf-236f-4d9c-86ca-ad0317f6a331","Type":"ContainerStarted","Data":"3dbff0c881d2d594f79f9946e3a3fc21163767bfed3ff1c3410679d3bdbcd0d3"} Nov 02 18:52:51.669220 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.668852 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" event={"ID":"cce9c5cf-236f-4d9c-86ca-ad0317f6a331","Type":"ContainerStarted","Data":"6ac2252bfcc5def4e720d81ebcfd0dc8d8615d491ab7e9118b16999d2ddc0517"} Nov 02 18:52:51.894488 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:51.894429 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podStartSLOduration=1.894414928 podStartE2EDuration="1.894414928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:52:51.842473021 +0000 UTC m=+5358.916000254" watchObservedRunningTime="2026-11-02 18:52:51.894414928 +0000 UTC m=+5358.967942131" Nov 02 18:52:52.655916 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:52.655881 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:52.657390 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:52.657363 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:52:52.657478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:52.657430 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 02 18:52:53.655946 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:53.655903 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:52:53.656355 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:53.655960 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 02 18:52:54.656459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:54.656422 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:52:54.657031 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:54.656478 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 02 18:52:55.656406 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:55.656365 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:52:55.656619 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:55.656424 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 02 18:52:56.656192 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:56.656155 5647 patch_prober.go:29] interesting pod/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" start-of-body= Nov 02 18:52:56.656390 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:56.656213 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.66:15021/healthz/ready\": dial tcp 10.134.1.66:15021: connect: connection refused" Nov 02 18:52:57.659908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:57.659872 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:57.660402 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:57.660332 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:52:57.661156 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:52:57.661135 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:53:05.657719 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:05.657683 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb"] Nov 02 18:53:05.658213 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:05.657904 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" containerID="cri-o://3dbff0c881d2d594f79f9946e3a3fc21163767bfed3ff1c3410679d3bdbcd0d3" gracePeriod=30 Nov 02 18:53:05.947897 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:53:05.947813 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:53:08.866623 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:08.866590 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dzrt7_069d21b8-f43f-465f-870f-1da8bfc01136/limitador/0.log" Nov 02 18:53:10.781956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:10.781923 5647 generic.go:356] "Generic (PLEG): container finished" podID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerID="3dbff0c881d2d594f79f9946e3a3fc21163767bfed3ff1c3410679d3bdbcd0d3" exitCode=0 Nov 02 18:53:10.782378 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:10.782000 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" event={"ID":"cce9c5cf-236f-4d9c-86ca-ad0317f6a331","Type":"ContainerDied","Data":"3dbff0c881d2d594f79f9946e3a3fc21163767bfed3ff1c3410679d3bdbcd0d3"} Nov 02 18:53:10.911076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:10.911048 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:53:11.065657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.065556 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.065657 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.065645 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.065951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.065701 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.065951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.065797 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.065951 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.065895 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.066142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.065971 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.066142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066017 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.066142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066022 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:53:11.066142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066041 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:53:11.066142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066064 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.066142 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066100 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy\") pod \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\" (UID: \"cce9c5cf-236f-4d9c-86ca-ad0317f6a331\") " Nov 02 18:53:11.066468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066180 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:53:11.066468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066271 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:53:11.066468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066282 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:53:11.066468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066438 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.066468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066457 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.066468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066473 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.066683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066481 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.066683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.066490 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.068281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.068257 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:53:11.068389 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.068332 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:53:11.068660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.068644 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "kube-api-access-2cptj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:53:11.068789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.068769 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy" pod "cce9c5cf-236f-4d9c-86ca-ad0317f6a331" (UID: "cce9c5cf-236f-4d9c-86ca-ad0317f6a331"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:53:11.168024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.167987 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.168024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.168016 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.168024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.168027 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cptj\" (UniqueName: \"kubernetes.io/projected/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-kube-api-access-2cptj\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.168024 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.168037 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce9c5cf-236f-4d9c-86ca-ad0317f6a331-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:53:11.384849 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.384812 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fd65b9f5-lqt67_3c9e8aee-a1e8-47ba-95fd-a7a5067d9367/manager/0.log" Nov 02 18:53:11.788589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.788506 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" Nov 02 18:53:11.789146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.788497 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb" event={"ID":"cce9c5cf-236f-4d9c-86ca-ad0317f6a331","Type":"ContainerDied","Data":"6ac2252bfcc5def4e720d81ebcfd0dc8d8615d491ab7e9118b16999d2ddc0517"} Nov 02 18:53:11.789146 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.788669 5647 scope.go:122] "RemoveContainer" containerID="3dbff0c881d2d594f79f9946e3a3fc21163767bfed3ff1c3410679d3bdbcd0d3" Nov 02 18:53:11.993649 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:11.993608 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb"] Nov 02 18:53:12.044427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:12.044342 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-6e3175-openshift-default-557ccfbfb5-jckcb"] Nov 02 18:53:12.088949 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:12.088915 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b6969_6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b/proxy/0.log" Nov 02 18:53:13.079862 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:13.079819 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" path="/var/lib/kubelet/pods/cce9c5cf-236f-4d9c-86ca-ad0317f6a331/volumes" Nov 02 18:53:17.023519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.023482 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp"] Nov 02 18:53:17.024022 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.024004 5647 cpu_manager.go:414] "containerMap: removing container" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" Nov 02 18:53:17.024092 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.024025 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" Nov 02 18:53:17.024135 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.024106 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce9c5cf-236f-4d9c-86ca-ad0317f6a331" containerName="istio-proxy" Nov 02 18:53:17.027190 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.027171 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.082683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.082650 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0604d6a-openshift-default-dockercfg-xtld5\"" Nov 02 18:53:17.127114 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127077 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6lc\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127124 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127293 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127349 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127386 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127471 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127432 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127599 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127546 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127598 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.127647 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.127627 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.134154 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.134129 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp"] Nov 02 18:53:17.228802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228766 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6lc\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.228802 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228811 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228847 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228878 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228905 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228933 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.228994 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229031 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229071 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229059 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229561 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229658 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229785 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229695 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229778 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.229910 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.229883 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.231715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.231693 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.231956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.231934 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.287144 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.287074 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.287291 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.287190 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6lc\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc\") pod \"e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.339960 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.339922 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:17.768983 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.768957 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp"] Nov 02 18:53:17.770333 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:53:17.770306 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91a9ff7_257f_4ecb_bc17_43ca31333034.slice/crio-721076bfecd322e019aaf5962ffa98de91deb8e605ad828b03a5729f9ef1a3da WatchSource:0}: Error finding container 721076bfecd322e019aaf5962ffa98de91deb8e605ad828b03a5729f9ef1a3da: Status 404 returned error can't find the container with id 721076bfecd322e019aaf5962ffa98de91deb8e605ad828b03a5729f9ef1a3da Nov 02 18:53:17.772345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.772311 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:53:17.772408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.772380 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:53:17.772447 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.772412 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:53:17.821548 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:17.821516 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" event={"ID":"a91a9ff7-257f-4ecb-bc17-43ca31333034","Type":"ContainerStarted","Data":"721076bfecd322e019aaf5962ffa98de91deb8e605ad828b03a5729f9ef1a3da"} Nov 02 18:53:18.828850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:18.828811 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" event={"ID":"a91a9ff7-257f-4ecb-bc17-43ca31333034","Type":"ContainerStarted","Data":"60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84"} Nov 02 18:53:18.998464 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:18.998401 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podStartSLOduration=2.998382726 podStartE2EDuration="2.998382726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:18.99738563 +0000 UTC m=+5386.070912844" watchObservedRunningTime="2026-11-02 18:53:18.998382726 +0000 UTC m=+5386.071909929" Nov 02 18:53:19.340438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:19.340397 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:19.345401 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:19.345379 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:19.834309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:19.834220 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:19.835171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:19.835153 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:24.758422 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.758391 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6"] Nov 02 18:53:24.762046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.762024 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:24.804571 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.804535 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cv7\" (UniqueName: \"kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7\") pod \"payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:24.804817 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.804595 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:24.818536 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.818503 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6"] Nov 02 18:53:24.866198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.866171 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a0604d6a-dockercfg-frpv9\"" Nov 02 18:53:24.866198 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.866170 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a0604d6a\"" Nov 02 18:53:24.905352 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.905318 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cv7\" (UniqueName: \"kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7\") pod \"payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:24.905515 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.905373 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:24.906095 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.906075 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume\") pod \"payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:24.962871 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:24.962840 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cv7\" (UniqueName: \"kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7\") pod \"payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:25.075692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:25.075605 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:25.450469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:25.450443 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6"] Nov 02 18:53:25.452372 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:53:25.452343 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4cf1025_f41d_43d7_938f_5510f22c95d2.slice/crio-17b142007613383b3a282e89688d480c29c586a97c98a0aa209cf04192245a79 WatchSource:0}: Error finding container 17b142007613383b3a282e89688d480c29c586a97c98a0aa209cf04192245a79: Status 404 returned error can't find the container with id 17b142007613383b3a282e89688d480c29c586a97c98a0aa209cf04192245a79 Nov 02 18:53:25.867096 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:25.867062 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" event={"ID":"a4cf1025-f41d-43d7-938f-5510f22c95d2","Type":"ContainerStarted","Data":"0f64cd65c54ca13f17cae4377fe9fd987868b866942d82e22f8d265722fc2607"} Nov 02 18:53:25.867616 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:25.867123 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" event={"ID":"a4cf1025-f41d-43d7-938f-5510f22c95d2","Type":"ContainerStarted","Data":"17b142007613383b3a282e89688d480c29c586a97c98a0aa209cf04192245a79"} Nov 02 18:53:25.867616 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:25.867156 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:25.983616 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:25.983557 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" podStartSLOduration=1.983543123 podStartE2EDuration="1.983543123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:53:25.982654022 +0000 UTC m=+5393.056181226" watchObservedRunningTime="2026-11-02 18:53:25.983543123 +0000 UTC m=+5393.057070325" Nov 02 18:53:34.900441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:34.900405 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:53:34.972857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:34.972829 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:53:34.979088 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:34.979066 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:53:35.953541 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:53:35.953501 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:53:36.879214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:36.879181 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:53:37.924578 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.924412 5647 scope.go:122] "RemoveContainer" containerID="5fd54c2ff903365aaa8603fe1a699a1792579b008d4299ec74e90bd121666639" Nov 02 18:53:37.937104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.937079 5647 scope.go:122] "RemoveContainer" containerID="1481554347aa3603216b412e77e035d4086f6c1606fa55a8ad03b60d54603d62" Nov 02 18:53:37.946237 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.946213 5647 scope.go:122] "RemoveContainer" containerID="6231f704c4917a866b2ae120f5d09e533a5f6f76a02bb63cd7077c1cd70ad9fa" Nov 02 18:53:37.954656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.954631 5647 scope.go:122] "RemoveContainer" containerID="3c3a63257536a5aef4ca6d7a4379a3b340e821ff978110d058792de8c4d164a3" Nov 02 18:53:37.963249 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.963229 5647 scope.go:122] "RemoveContainer" containerID="90df631a2581ef8858b5f2063e45c71c0f85e8f83d1e5fc0d4152c63237758f9" Nov 02 18:53:37.975679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.975656 5647 scope.go:122] "RemoveContainer" containerID="dceb129272a174bb230887b70246150cb0d34fd362280765a061077a950d1580" Nov 02 18:53:37.984316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.984292 5647 scope.go:122] "RemoveContainer" containerID="3dd07989c1ffa78ff2e840372a67a64b334c2c3b9953b27608ff2cd9ba711820" Nov 02 18:53:37.992693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:37.992668 5647 scope.go:122] "RemoveContainer" containerID="f3b65b5a870622c1bd12bff8c1e1b9fe172372d574bb14c790b01b865d7593d9" Nov 02 18:53:44.983577 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:44.983540 5647 generic.go:356] "Generic (PLEG): container finished" podID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerID="60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84" exitCode=0 Nov 02 18:53:44.984072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:44.983586 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" event={"ID":"a91a9ff7-257f-4ecb-bc17-43ca31333034","Type":"ContainerDied","Data":"60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84"} Nov 02 18:53:44.984072 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:44.983967 5647 scope.go:122] "RemoveContainer" containerID="60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84" Nov 02 18:53:44.985179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:44.985148 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:53:44.985301 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:44.985217 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:53:44.985301 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:44.985247 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:53:45.990963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:45.990920 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" event={"ID":"a91a9ff7-257f-4ecb-bc17-43ca31333034","Type":"ContainerStarted","Data":"ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d"} Nov 02 18:53:46.340456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:46.340421 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:46.341547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:46.341523 5647 patch_prober.go:29] interesting pod/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" start-of-body= Nov 02 18:53:46.341608 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:46.341575 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" Nov 02 18:53:47.340866 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:47.340821 5647 patch_prober.go:29] interesting pod/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" start-of-body= Nov 02 18:53:47.341293 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:47.340899 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" Nov 02 18:53:48.341104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:48.341062 5647 patch_prober.go:29] interesting pod/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" start-of-body= Nov 02 18:53:48.341533 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:48.341136 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" Nov 02 18:53:49.340808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:49.340773 5647 patch_prober.go:29] interesting pod/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" start-of-body= Nov 02 18:53:49.341015 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:49.340828 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" Nov 02 18:53:49.834498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:49.834404 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:50.340491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:50.340452 5647 patch_prober.go:29] interesting pod/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" start-of-body= Nov 02 18:53:50.340755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:50.340528 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.67:15021/healthz/ready\": dial tcp 10.134.1.67:15021: connect: connection refused" Nov 02 18:53:51.344910 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:51.344877 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:51.345888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:51.345870 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:53:56.047689 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:56.047638 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw"] Nov 02 18:53:56.098155 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:56.098114 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a69ce2e2-hwcdw"] Nov 02 18:53:57.079032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.078998 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1" path="/var/lib/kubelet/pods/9ca210fb-e2b1-4d5f-81aa-4e63ba7451d1/volumes" Nov 02 18:53:57.536331 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.536294 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68"] Nov 02 18:53:57.540052 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.540035 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.645698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.645661 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:53:57.645970 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.645715 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:53:57.647679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.647658 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68"] Nov 02 18:53:57.652227 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.652207 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.652362 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.652343 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.652435 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.652407 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxbm\" (UniqueName: \"kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.752973 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.752935 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxbm\" (UniqueName: \"kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.753178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.752995 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.753178 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.753110 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.753697 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.753673 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.755641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.755618 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.811777 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.811688 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxbm\" (UniqueName: \"kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm\") pod \"maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:57.852957 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:57.852925 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:53:58.221806 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:58.221782 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68"] Nov 02 18:53:58.223597 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:53:58.223570 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24f58d8_7f90_41d5_af0f_e597419ac17a.slice/crio-232e0010a4c79fd26ed220255820f3b90aeca04a152ddbaf405b4da0c0f21494 WatchSource:0}: Error finding container 232e0010a4c79fd26ed220255820f3b90aeca04a152ddbaf405b4da0c0f21494: Status 404 returned error can't find the container with id 232e0010a4c79fd26ed220255820f3b90aeca04a152ddbaf405b4da0c0f21494 Nov 02 18:53:59.081588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:59.081556 5647 generic.go:356] "Generic (PLEG): container finished" podID="d24f58d8-7f90-41d5-af0f-e597419ac17a" containerID="196b52d1c4fb46114f81cc196648d925017ce90ddf4db71ec88716b5d6563035" exitCode=0 Nov 02 18:53:59.081776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:59.081637 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" event={"ID":"d24f58d8-7f90-41d5-af0f-e597419ac17a","Type":"ContainerDied","Data":"196b52d1c4fb46114f81cc196648d925017ce90ddf4db71ec88716b5d6563035"} Nov 02 18:53:59.081776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:53:59.081694 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" event={"ID":"d24f58d8-7f90-41d5-af0f-e597419ac17a","Type":"ContainerStarted","Data":"232e0010a4c79fd26ed220255820f3b90aeca04a152ddbaf405b4da0c0f21494"} Nov 02 18:54:00.222119 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.222091 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:54:00.278202 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.278171 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca\") pod \"d24f58d8-7f90-41d5-af0f-e597419ac17a\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " Nov 02 18:54:00.278366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.278349 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca\") pod \"d24f58d8-7f90-41d5-af0f-e597419ac17a\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " Nov 02 18:54:00.278420 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.278389 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm\" (UniqueName: \"kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm\") pod \"d24f58d8-7f90-41d5-af0f-e597419ac17a\" (UID: \"d24f58d8-7f90-41d5-af0f-e597419ac17a\") " Nov 02 18:54:00.278588 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.278569 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca" pod "d24f58d8-7f90-41d5-af0f-e597419ac17a" (UID: "d24f58d8-7f90-41d5-af0f-e597419ac17a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:00.278698 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.278678 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:00.280585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.280554 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca" pod "d24f58d8-7f90-41d5-af0f-e597419ac17a" (UID: "d24f58d8-7f90-41d5-af0f-e597419ac17a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:00.280682 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.280580 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm" pod "d24f58d8-7f90-41d5-af0f-e597419ac17a" (UID: "d24f58d8-7f90-41d5-af0f-e597419ac17a"). InnerVolumeSpecName "kube-api-access-lkxbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:00.379398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.379318 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24f58d8-7f90-41d5-af0f-e597419ac17a-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:00.379398 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:00.379346 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxbm\" (UniqueName: \"kubernetes.io/projected/d24f58d8-7f90-41d5-af0f-e597419ac17a-kube-api-access-lkxbm\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:01.092413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:01.092376 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" event={"ID":"d24f58d8-7f90-41d5-af0f-e597419ac17a","Type":"ContainerDied","Data":"232e0010a4c79fd26ed220255820f3b90aeca04a152ddbaf405b4da0c0f21494"} Nov 02 18:54:01.092413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:01.092400 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68" Nov 02 18:54:01.092612 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:01.092429 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232e0010a4c79fd26ed220255820f3b90aeca04a152ddbaf405b4da0c0f21494" Nov 02 18:54:05.946359 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:54:05.946320 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:54:09.024065 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:09.024026 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6"] Nov 02 18:54:13.043904 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:13.043861 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz"] Nov 02 18:54:13.094911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:13.094878 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz"] Nov 02 18:54:13.128635 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:54:13.128595 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-da24afb7-94xnz\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da24afb7-94xnz" Nov 02 18:54:14.024456 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.024399 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" podUID="a4cf1025-f41d-43d7-938f-5510f22c95d2" containerName="payload-processing" containerID="cri-o://0f64cd65c54ca13f17cae4377fe9fd987868b866942d82e22f8d265722fc2607" gracePeriod=25 Nov 02 18:54:14.161655 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.161623 5647 generic.go:356] "Generic (PLEG): container finished" podID="a4cf1025-f41d-43d7-938f-5510f22c95d2" containerID="0f64cd65c54ca13f17cae4377fe9fd987868b866942d82e22f8d265722fc2607" exitCode=0 Nov 02 18:54:14.162077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.161693 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" event={"ID":"a4cf1025-f41d-43d7-938f-5510f22c95d2","Type":"ContainerDied","Data":"0f64cd65c54ca13f17cae4377fe9fd987868b866942d82e22f8d265722fc2607"} Nov 02 18:54:14.281601 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.281526 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:54:14.413373 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.413330 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7\" (UniqueName: \"kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7\") pod \"a4cf1025-f41d-43d7-938f-5510f22c95d2\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " Nov 02 18:54:14.413551 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.413381 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume\") pod \"a4cf1025-f41d-43d7-938f-5510f22c95d2\" (UID: \"a4cf1025-f41d-43d7-938f-5510f22c95d2\") " Nov 02 18:54:14.413789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.413763 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume" pod "a4cf1025-f41d-43d7-938f-5510f22c95d2" (UID: "a4cf1025-f41d-43d7-938f-5510f22c95d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:14.415603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.415580 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7" pod "a4cf1025-f41d-43d7-938f-5510f22c95d2" (UID: "a4cf1025-f41d-43d7-938f-5510f22c95d2"). InnerVolumeSpecName "kube-api-access-k9cv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:14.515066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.515022 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9cv7\" (UniqueName: \"kubernetes.io/projected/a4cf1025-f41d-43d7-938f-5510f22c95d2-kube-api-access-k9cv7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:14.515066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:14.515057 5647 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4cf1025-f41d-43d7-938f-5510f22c95d2-plugins-config-volume\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:15.079541 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:15.079507 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2" path="/var/lib/kubelet/pods/8be5d81b-ba54-46fd-9c5e-e40ec23f9aa2/volumes" Nov 02 18:54:15.168684 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:15.168653 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" Nov 02 18:54:15.169121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:15.168658 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6" event={"ID":"a4cf1025-f41d-43d7-938f-5510f22c95d2","Type":"ContainerDied","Data":"17b142007613383b3a282e89688d480c29c586a97c98a0aa209cf04192245a79"} Nov 02 18:54:15.169121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:15.168808 5647 scope.go:122] "RemoveContainer" containerID="0f64cd65c54ca13f17cae4377fe9fd987868b866942d82e22f8d265722fc2607" Nov 02 18:54:15.335705 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:15.335601 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6"] Nov 02 18:54:15.385856 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:15.385822 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a0604d6a-55cc6c75c9-8bfs6"] Nov 02 18:54:16.546570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:16.546538 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68"] Nov 02 18:54:16.582287 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:16.582259 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a0604d6a-2qw68"] Nov 02 18:54:17.080901 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:17.080868 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4cf1025-f41d-43d7-938f-5510f22c95d2" path="/var/lib/kubelet/pods/a4cf1025-f41d-43d7-938f-5510f22c95d2/volumes" Nov 02 18:54:17.081276 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:17.081262 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24f58d8-7f90-41d5-af0f-e597419ac17a" path="/var/lib/kubelet/pods/d24f58d8-7f90-41d5-af0f-e597419ac17a/volumes" Nov 02 18:54:18.629131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:18.629100 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp"] Nov 02 18:54:18.629614 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:18.629319 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" containerID="cri-o://ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d" gracePeriod=30 Nov 02 18:54:19.049047 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:19.048963 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:54:21.347529 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:21.347487 5647 patch_prober.go:29] interesting pod/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 18:54:21.347959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:21.347557 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 18:54:22.983835 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:22.983794 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-lqt67"] Nov 02 18:54:22.984377 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:22.984164 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" podUID="3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" containerName="manager" containerID="cri-o://72780a2c28595b9f97e42fdb8b1ba15a1d76fb1c9552c287471e1b817f24de45" gracePeriod=30 Nov 02 18:54:23.219214 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.219176 5647 generic.go:356] "Generic (PLEG): container finished" podID="3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" containerID="72780a2c28595b9f97e42fdb8b1ba15a1d76fb1c9552c287471e1b817f24de45" exitCode=0 Nov 02 18:54:23.219363 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.219216 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" event={"ID":"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367","Type":"ContainerDied","Data":"72780a2c28595b9f97e42fdb8b1ba15a1d76fb1c9552c287471e1b817f24de45"} Nov 02 18:54:23.239755 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.239676 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 18:54:23.296032 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.296000 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs\") pod \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " Nov 02 18:54:23.296207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.296039 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert\") pod \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " Nov 02 18:54:23.296207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.296077 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j\" (UniqueName: \"kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j\") pod \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\" (UID: \"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367\") " Nov 02 18:54:23.298314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.298279 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs" pod "3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" (UID: "3c9e8aee-a1e8-47ba-95fd-a7a5067d9367"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:23.298452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.298393 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j" pod "3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" (UID: "3c9e8aee-a1e8-47ba-95fd-a7a5067d9367"). InnerVolumeSpecName "kube-api-access-7p62j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:23.298507 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.298450 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert" pod "3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" (UID: "3c9e8aee-a1e8-47ba-95fd-a7a5067d9367"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:23.397026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.396996 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:23.397026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.397021 5647 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-webhook-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:23.397026 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.397032 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p62j\" (UniqueName: \"kubernetes.io/projected/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367-kube-api-access-7p62j\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:23.872215 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.872189 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:54:23.901466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901370 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901415 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901466 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901457 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901481 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901508 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901526 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901598 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901648 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901674 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs\") pod \"a91a9ff7-257f-4ecb-bc17-43ca31333034\" (UID: \"a91a9ff7-257f-4ecb-bc17-43ca31333034\") " Nov 02 18:54:23.901888 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901830 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:54:23.902486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.901885 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:54:23.902486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.902055 5647 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-data\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:23.902486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.902068 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:54:23.902486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.902075 5647 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-credential-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:23.902486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.902093 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:23.902486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.902200 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:54:23.903805 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.903778 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 18:54:23.903945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.903923 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 18:54:23.904009 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.903985 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "kube-api-access-7d6lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:23.904125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:23.904111 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token" pod "a91a9ff7-257f-4ecb-bc17-43ca31333034" (UID: "a91a9ff7-257f-4ecb-bc17-43ca31333034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:24.002989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.002949 5647 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-podinfo\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.002989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.002977 5647 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-token\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.002989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.002986 5647 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.002989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.002994 5647 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-istio-envoy\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.002989 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.003003 5647 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91a9ff7-257f-4ecb-bc17-43ca31333034-istiod-ca-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.003772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.003013 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6lc\" (UniqueName: \"kubernetes.io/projected/a91a9ff7-257f-4ecb-bc17-43ca31333034-kube-api-access-7d6lc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.003772 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.003022 5647 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91a9ff7-257f-4ecb-bc17-43ca31333034-workload-socket\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:24.226312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.226227 5647 generic.go:356] "Generic (PLEG): container finished" podID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerID="ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d" exitCode=0 Nov 02 18:54:24.226312 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.226276 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" Nov 02 18:54:24.226538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.226309 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" event={"ID":"a91a9ff7-257f-4ecb-bc17-43ca31333034","Type":"ContainerDied","Data":"ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d"} Nov 02 18:54:24.226538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.226359 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp" event={"ID":"a91a9ff7-257f-4ecb-bc17-43ca31333034","Type":"ContainerDied","Data":"721076bfecd322e019aaf5962ffa98de91deb8e605ad828b03a5729f9ef1a3da"} Nov 02 18:54:24.226538 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.226384 5647 scope.go:122] "RemoveContainer" containerID="ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d" Nov 02 18:54:24.227826 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.227804 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" event={"ID":"3c9e8aee-a1e8-47ba-95fd-a7a5067d9367","Type":"ContainerDied","Data":"385c2eb65f792aa1d69bccd4a574e2712d2b58b292ee5bef4c7d1c12f933d04d"} Nov 02 18:54:24.227898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.227819 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-lqt67" Nov 02 18:54:24.236984 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.236960 5647 scope.go:122] "RemoveContainer" containerID="60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84" Nov 02 18:54:24.245813 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.245788 5647 scope.go:122] "RemoveContainer" containerID="ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d" Nov 02 18:54:24.246125 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:54:24.246106 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d\": container with ID starting with ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d not found: ID does not exist" containerID="ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d" Nov 02 18:54:24.246177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.246136 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d"} err="failed to get container status \"ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d\": rpc error: code = NotFound desc = could not find container \"ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d\": container with ID starting with ce2466836decba71c01509109264627c97fabe7ddec4290b04aa51eb1d5fe33d not found: ID does not exist" Nov 02 18:54:24.246177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.246155 5647 scope.go:122] "RemoveContainer" containerID="60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84" Nov 02 18:54:24.246480 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:54:24.246459 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84\": container with ID starting with 60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84 not found: ID does not exist" containerID="60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84" Nov 02 18:54:24.246553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.246491 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84"} err="failed to get container status \"60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84\": rpc error: code = NotFound desc = could not find container \"60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84\": container with ID starting with 60f5cb6dd2de00cd2c818559152ecf131e5b962572a5697e9ad9a34ac98baa84 not found: ID does not exist" Nov 02 18:54:24.246553 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.246515 5647 scope.go:122] "RemoveContainer" containerID="72780a2c28595b9f97e42fdb8b1ba15a1d76fb1c9552c287471e1b817f24de45" Nov 02 18:54:24.392860 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.392827 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp"] Nov 02 18:54:24.443627 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.443586 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0604d6a-openshift-default-79f9b46c97-scnzp"] Nov 02 18:54:24.549101 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.549017 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-lqt67"] Nov 02 18:54:24.554587 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:24.554561 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-lqt67"] Nov 02 18:54:25.079437 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.079403 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" path="/var/lib/kubelet/pods/3c9e8aee-a1e8-47ba-95fd-a7a5067d9367/volumes" Nov 02 18:54:25.079884 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.079852 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" path="/var/lib/kubelet/pods/a91a9ff7-257f-4ecb-bc17-43ca31333034/volumes" Nov 02 18:54:25.377574 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.377542 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-d5jfh"] Nov 02 18:54:25.377953 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.377939 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" Nov 02 18:54:25.378010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.377956 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" Nov 02 18:54:25.378010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.377975 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" containerName="manager" Nov 02 18:54:25.378010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.377981 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" containerName="manager" Nov 02 18:54:25.378010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.377995 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d24f58d8-7f90-41d5-af0f-e597419ac17a" containerName="revoke-keys" Nov 02 18:54:25.378010 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378001 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24f58d8-7f90-41d5-af0f-e597419ac17a" containerName="revoke-keys" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378021 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a4cf1025-f41d-43d7-938f-5510f22c95d2" containerName="payload-processing" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378026 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4cf1025-f41d-43d7-938f-5510f22c95d2" containerName="payload-processing" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378084 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24f58d8-7f90-41d5-af0f-e597419ac17a" containerName="revoke-keys" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378092 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378100 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4cf1025-f41d-43d7-938f-5510f22c95d2" containerName="payload-processing" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378107 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" Nov 02 18:54:25.378186 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.378114 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9e8aee-a1e8-47ba-95fd-a7a5067d9367" containerName="manager" Nov 02 18:54:25.382834 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.382812 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.415684 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.415647 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.415898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.415718 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.415898 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.415844 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjrb\" (UniqueName: \"kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.436195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.436160 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:54:25.487125 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.487094 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tk2kj\"" Nov 02 18:54:25.487328 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.487244 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:54:25.487385 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.487334 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:54:25.489068 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.489046 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-d5jfh"] Nov 02 18:54:25.516937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.516895 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.517105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.516972 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.517105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.517005 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjrb\" (UniqueName: \"kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.519530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.519498 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.519680 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.519552 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.584274 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.584238 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjrb\" (UniqueName: \"kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb\") pod \"maas-controller-7fd65b9f5-d5jfh\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:25.696251 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:25.696163 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:26.070625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:26.070599 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-d5jfh"] Nov 02 18:54:26.072622 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:54:26.072594 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bd733c_cef8_429c_98ea_5c3058bef0bb.slice/crio-e6ec114f4607207c0a22c904c342d1001d5ef30ad0b322e8bfde7a58364acb4f WatchSource:0}: Error finding container e6ec114f4607207c0a22c904c342d1001d5ef30ad0b322e8bfde7a58364acb4f: Status 404 returned error can't find the container with id e6ec114f4607207c0a22c904c342d1001d5ef30ad0b322e8bfde7a58364acb4f Nov 02 18:54:26.074059 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:26.074041 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 18:54:26.241480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:26.241448 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" event={"ID":"d7bd733c-cef8-429c-98ea-5c3058bef0bb","Type":"ContainerStarted","Data":"e6ec114f4607207c0a22c904c342d1001d5ef30ad0b322e8bfde7a58364acb4f"} Nov 02 18:54:27.248157 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:27.248119 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" event={"ID":"d7bd733c-cef8-429c-98ea-5c3058bef0bb","Type":"ContainerStarted","Data":"3009519295f7989eee9785a949e49e12ca157df955492e8a6df1424ed6cf6d53"} Nov 02 18:54:27.248575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:27.248185 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:27.386425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:27.386368 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" podStartSLOduration=1.409315159 podStartE2EDuration="2.386353711s" totalImagesPullingTime="977.038552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:54:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:54:27.384968258 +0000 UTC m=+5454.458495462" watchObservedRunningTime="2026-11-02 18:54:27.386353711 +0000 UTC m=+5454.459880914" Nov 02 18:54:35.947268 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:54:35.947212 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:54:38.044275 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:38.044240 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp"] Nov 02 18:54:38.094854 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:38.094830 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650d02a1-8bkfp"] Nov 02 18:54:38.255002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:38.254973 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:39.078997 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:39.078965 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6693ff-12fc-4ce5-a0cc-53609f012ee0" path="/var/lib/kubelet/pods/ed6693ff-12fc-4ce5-a0cc-53609f012ee0/volumes" Nov 02 18:54:46.585597 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.585559 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj"] Nov 02 18:54:46.586241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.586221 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" Nov 02 18:54:46.586296 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.586245 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91a9ff7-257f-4ecb-bc17-43ca31333034" containerName="istio-proxy" Nov 02 18:54:46.589618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.589592 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.644303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.644270 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj"] Nov 02 18:54:46.693174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.693145 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:54:46.693174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.693145 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:54:46.716346 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.716304 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dt7h\" (UniqueName: \"kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.716521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.716427 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.716521 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.716499 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.817954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.817913 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dt7h\" (UniqueName: \"kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.818175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.817980 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.818175 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.818005 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.819016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.818860 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.820678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.820653 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.876206 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.876177 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dt7h\" (UniqueName: \"kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h\") pod \"maas-api-revoke-sub-85237e45-4rdtj\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:46.902007 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:46.901980 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:47.272525 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:47.272474 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj"] Nov 02 18:54:47.273758 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:54:47.273701 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa6939d_13a0_4ceb_bc3d_4f7f60ac55fd.slice/crio-3cd0c91336e464671c03490cf8a5ac2547c8918ab5174e4bd14f06c3377dd12a WatchSource:0}: Error finding container 3cd0c91336e464671c03490cf8a5ac2547c8918ab5174e4bd14f06c3377dd12a: Status 404 returned error can't find the container with id 3cd0c91336e464671c03490cf8a5ac2547c8918ab5174e4bd14f06c3377dd12a Nov 02 18:54:47.364451 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:47.364414 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" event={"ID":"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd","Type":"ContainerStarted","Data":"3cd0c91336e464671c03490cf8a5ac2547c8918ab5174e4bd14f06c3377dd12a"} Nov 02 18:54:48.371491 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:48.371448 5647 generic.go:356] "Generic (PLEG): container finished" podID="7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" containerID="8850ce0955f59d21a0c9fbca357db98c9a563328080bbafad1bad156151c5451" exitCode=0 Nov 02 18:54:48.371962 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:48.371537 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" event={"ID":"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd","Type":"ContainerDied","Data":"8850ce0955f59d21a0c9fbca357db98c9a563328080bbafad1bad156151c5451"} Nov 02 18:54:49.505945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.505920 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:49.648531 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.648455 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h\" (UniqueName: \"kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h\") pod \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " Nov 02 18:54:49.648531 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.648512 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca\") pod \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " Nov 02 18:54:49.648750 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.648597 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca\") pod \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\" (UID: \"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd\") " Nov 02 18:54:49.648963 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.648933 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca" pod "7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" (UID: "7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:54:49.650844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.650816 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca" pod "7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" (UID: "7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:49.650996 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.650840 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h" pod "7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" (UID: "7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd"). InnerVolumeSpecName "kube-api-access-4dt7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:49.749712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.749662 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:49.749712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.749708 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dt7h\" (UniqueName: \"kubernetes.io/projected/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-kube-api-access-4dt7h\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:49.749712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:49.749755 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:50.383956 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:50.383919 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" Nov 02 18:54:50.384165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:50.383970 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj" event={"ID":"7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd","Type":"ContainerDied","Data":"3cd0c91336e464671c03490cf8a5ac2547c8918ab5174e4bd14f06c3377dd12a"} Nov 02 18:54:50.384165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:50.384019 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd0c91336e464671c03490cf8a5ac2547c8918ab5174e4bd14f06c3377dd12a" Nov 02 18:54:53.001852 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.001815 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb"] Nov 02 18:54:53.002261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.002213 5647 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" containerName="revoke-keys" Nov 02 18:54:53.002261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.002224 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" containerName="revoke-keys" Nov 02 18:54:53.002360 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.002304 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" containerName="revoke-keys" Nov 02 18:54:53.005241 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.005225 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.052659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.052629 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j"] Nov 02 18:54:53.054105 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.054085 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c63887-s5d4j"] Nov 02 18:54:53.080152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.080120 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e8691b-3138-4ea4-975f-3a341eeb9c3e" path="/var/lib/kubelet/pods/64e8691b-3138-4ea4-975f-3a341eeb9c3e/volumes" Nov 02 18:54:53.082284 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082262 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvxf5\" (UniqueName: \"kubernetes.io/projected/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-kube-api-access-rvxf5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082305 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082386 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082377 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082399 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082518 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082486 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082514 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082565 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082542 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082646 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082604 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.082685 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.082659 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.101842 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.101803 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v7x48\"" Nov 02 18:54:53.103840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.103810 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb"] Nov 02 18:54:53.184091 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184055 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184091 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184091 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvxf5\" (UniqueName: \"kubernetes.io/projected/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-kube-api-access-rvxf5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184122 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184275 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184320 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184305 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184355 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184387 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184469 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184438 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184482 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184495 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184641 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184570 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184852 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184837 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.184924 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.184900 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.185055 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.185021 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.186815 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.186793 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.187002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.186984 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.243678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.243644 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.243678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.243672 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvxf5\" (UniqueName: \"kubernetes.io/projected/7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607-kube-api-access-rvxf5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-26plb\" (UID: \"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.317691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.317612 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:53.740681 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.740638 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb"] Nov 02 18:54:53.742485 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:54:53.742454 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebbc3ec_4bd5_40fb_8ce6_338f3b4b7607.slice/crio-e0c7c039cc558f9f41a83275608079f4e75b1accb93cabcb7b8f0a7a71738074 WatchSource:0}: Error finding container e0c7c039cc558f9f41a83275608079f4e75b1accb93cabcb7b8f0a7a71738074: Status 404 returned error can't find the container with id e0c7c039cc558f9f41a83275608079f4e75b1accb93cabcb7b8f0a7a71738074 Nov 02 18:54:53.744692 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.744656 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:54:53.744789 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.744757 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:54:53.744844 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:53.744797 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:54:54.431653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:54.431610 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" event={"ID":"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607","Type":"ContainerStarted","Data":"370e3852d91e12cb1afa98e62d8e3b989cc3c7e0f8d4dd9a32ea33c8059246dd"} Nov 02 18:54:54.432104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:54.431696 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" event={"ID":"7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607","Type":"ContainerStarted","Data":"e0c7c039cc558f9f41a83275608079f4e75b1accb93cabcb7b8f0a7a71738074"} Nov 02 18:54:54.551688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:54.551635 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podStartSLOduration=2.551622731 podStartE2EDuration="2.551622731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:54:54.549841375 +0000 UTC m=+5481.623368578" watchObservedRunningTime="2026-11-02 18:54:54.551622731 +0000 UTC m=+5481.625149933" Nov 02 18:54:55.321837 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:55.321799 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:54:55.322776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:55.322717 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 02 18:54:55.322937 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:55.322805 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 02 18:54:56.319760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:56.318151 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 02 18:54:56.319760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:56.318229 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 02 18:54:57.324761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:57.324226 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 02 18:54:57.324761 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:57.324287 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 02 18:54:58.322760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:58.322212 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 02 18:54:58.322760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:58.322282 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 02 18:54:59.198253 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.198213 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-d5jfh"] Nov 02 18:54:59.198796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.198465 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" podUID="d7bd733c-cef8-429c-98ea-5c3058bef0bb" containerName="manager" containerID="cri-o://3009519295f7989eee9785a949e49e12ca157df955492e8a6df1424ed6cf6d53" gracePeriod=30 Nov 02 18:54:59.326763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.323121 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 02 18:54:59.326763 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.323184 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 02 18:54:59.462552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.461725 5647 generic.go:356] "Generic (PLEG): container finished" podID="d7bd733c-cef8-429c-98ea-5c3058bef0bb" containerID="3009519295f7989eee9785a949e49e12ca157df955492e8a6df1424ed6cf6d53" exitCode=0 Nov 02 18:54:59.462552 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.461780 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" event={"ID":"d7bd733c-cef8-429c-98ea-5c3058bef0bb","Type":"ContainerDied","Data":"3009519295f7989eee9785a949e49e12ca157df955492e8a6df1424ed6cf6d53"} Nov 02 18:54:59.482715 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.482690 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:54:59.571270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.571233 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb\" (UniqueName: \"kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb\") pod \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " Nov 02 18:54:59.571459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.571354 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert\") pod \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " Nov 02 18:54:59.571459 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.571391 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs\") pod \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\" (UID: \"d7bd733c-cef8-429c-98ea-5c3058bef0bb\") " Nov 02 18:54:59.573618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.573579 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs" pod "d7bd733c-cef8-429c-98ea-5c3058bef0bb" (UID: "d7bd733c-cef8-429c-98ea-5c3058bef0bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:59.573618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.573599 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert" pod "d7bd733c-cef8-429c-98ea-5c3058bef0bb" (UID: "d7bd733c-cef8-429c-98ea-5c3058bef0bb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:54:59.573618 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.573600 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb" pod "d7bd733c-cef8-429c-98ea-5c3058bef0bb" (UID: "d7bd733c-cef8-429c-98ea-5c3058bef0bb"). InnerVolumeSpecName "kube-api-access-9tjrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:54:59.672569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.672531 5647 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-webhook-cert\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:59.672569 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.672566 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7bd733c-cef8-429c-98ea-5c3058bef0bb-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:54:59.672838 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:54:59.672580 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjrb\" (UniqueName: \"kubernetes.io/projected/d7bd733c-cef8-429c-98ea-5c3058bef0bb-kube-api-access-9tjrb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:55:00.045885 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.045846 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd"] Nov 02 18:55:00.047396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.047369 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-527e8f4e-28rfd"] Nov 02 18:55:00.318676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.318577 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 02 18:55:00.318676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.318649 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 02 18:55:00.477930 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.477897 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" Nov 02 18:55:00.478152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.477923 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-d5jfh" event={"ID":"d7bd733c-cef8-429c-98ea-5c3058bef0bb","Type":"ContainerDied","Data":"e6ec114f4607207c0a22c904c342d1001d5ef30ad0b322e8bfde7a58364acb4f"} Nov 02 18:55:00.478152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.478022 5647 scope.go:122] "RemoveContainer" containerID="3009519295f7989eee9785a949e49e12ca157df955492e8a6df1424ed6cf6d53" Nov 02 18:55:00.643519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.643486 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-d5jfh"] Nov 02 18:55:00.694021 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:00.693985 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-d5jfh"] Nov 02 18:55:01.079720 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:01.079635 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac824068-ece7-410e-975c-6493ee530557" path="/var/lib/kubelet/pods/ac824068-ece7-410e-975c-6493ee530557/volumes" Nov 02 18:55:01.080061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:01.080047 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7bd733c-cef8-429c-98ea-5c3058bef0bb" path="/var/lib/kubelet/pods/d7bd733c-cef8-429c-98ea-5c3058bef0bb/volumes" Nov 02 18:55:02.016183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.016147 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-t84fb"] Nov 02 18:55:02.016606 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.016555 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d7bd733c-cef8-429c-98ea-5c3058bef0bb" containerName="manager" Nov 02 18:55:02.016606 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.016572 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bd733c-cef8-429c-98ea-5c3058bef0bb" containerName="manager" Nov 02 18:55:02.016687 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.016670 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bd733c-cef8-429c-98ea-5c3058bef0bb" containerName="manager" Nov 02 18:55:02.021189 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.021167 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.090292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.090254 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:55:02.090724 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.090700 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:55:02.091514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.091494 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" Nov 02 18:55:02.094938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.094916 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0af76160-9759-4f5e-b760-4a7210a7c8c7-webhook-cert\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.094995 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.094969 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gcn\" (UniqueName: \"kubernetes.io/projected/0af76160-9759-4f5e-b760-4a7210a7c8c7-kube-api-access-s4gcn\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.095058 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.095038 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af76160-9759-4f5e-b760-4a7210a7c8c7-metrics-certs\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.125490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.125453 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 02 18:55:02.125490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.125486 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 02 18:55:02.125490 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.125462 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-tk2kj\"" Nov 02 18:55:02.125768 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.125515 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 02 18:55:02.127316 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.127292 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-t84fb"] Nov 02 18:55:02.196148 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.196115 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gcn\" (UniqueName: \"kubernetes.io/projected/0af76160-9759-4f5e-b760-4a7210a7c8c7-kube-api-access-s4gcn\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.196347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.196152 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af76160-9759-4f5e-b760-4a7210a7c8c7-metrics-certs\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.196347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.196263 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0af76160-9759-4f5e-b760-4a7210a7c8c7-webhook-cert\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.198875 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.198845 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0af76160-9759-4f5e-b760-4a7210a7c8c7-metrics-certs\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.198992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.198901 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0af76160-9759-4f5e-b760-4a7210a7c8c7-webhook-cert\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.254086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.254050 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gcn\" (UniqueName: \"kubernetes.io/projected/0af76160-9759-4f5e-b760-4a7210a7c8c7-kube-api-access-s4gcn\") pod \"maas-controller-7fd65b9f5-t84fb\" (UID: \"0af76160-9759-4f5e-b760-4a7210a7c8c7\") " pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.337118 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.337092 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:02.711012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:02.710982 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fd65b9f5-t84fb"] Nov 02 18:55:02.713468 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:55:02.713437 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af76160_9759_4f5e_b760_4a7210a7c8c7.slice/crio-cffd9f160086e9e9ddcaf59d801cf2a6e790be86e6b57871b5b5b2491f996b12 WatchSource:0}: Error finding container cffd9f160086e9e9ddcaf59d801cf2a6e790be86e6b57871b5b5b2491f996b12: Status 404 returned error can't find the container with id cffd9f160086e9e9ddcaf59d801cf2a6e790be86e6b57871b5b5b2491f996b12 Nov 02 18:55:03.495770 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:03.495720 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" event={"ID":"0af76160-9759-4f5e-b760-4a7210a7c8c7","Type":"ContainerStarted","Data":"cffd9f160086e9e9ddcaf59d801cf2a6e790be86e6b57871b5b5b2491f996b12"} Nov 02 18:55:04.503130 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:04.503092 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" event={"ID":"0af76160-9759-4f5e-b760-4a7210a7c8c7","Type":"ContainerStarted","Data":"62fa5364c23ef42f0fc0074aeb1ac9332bd120f04df72e91b7e1be982e803a90"} Nov 02 18:55:04.503530 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:04.503191 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:04.620074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:04.620017 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" podStartSLOduration=2.801328438 podStartE2EDuration="3.620000723s" totalImagesPullingTime="818.672285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:04.619507806 +0000 UTC m=+5491.693035005" watchObservedRunningTime="2026-11-02 18:55:04.620000723 +0000 UTC m=+5491.693527926" Nov 02 18:55:05.946985 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:55:05.946947 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:55:08.117061 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.117030 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8"] Nov 02 18:55:08.120787 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.120769 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.157585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157542 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.157796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157596 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237ddf20-334e-4c7e-8b07-6750aaa360b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.157796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157618 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.157796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157681 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.157796 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157783 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.158025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157829 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.158025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157856 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dnx\" (UniqueName: \"kubernetes.io/projected/237ddf20-334e-4c7e-8b07-6750aaa360b9-kube-api-access-v8dnx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.158025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157885 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.158025 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.157918 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.227368 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.227335 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8"] Nov 02 18:55:08.259089 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259052 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259102 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237ddf20-334e-4c7e-8b07-6750aaa360b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259124 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259148 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259172 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259216 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259242 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dnx\" (UniqueName: \"kubernetes.io/projected/237ddf20-334e-4c7e-8b07-6750aaa360b9-kube-api-access-v8dnx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259279 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259270 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259318 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259522 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.259625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259550 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.260149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259829 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.260149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259848 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.260149 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.259949 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/237ddf20-334e-4c7e-8b07-6750aaa360b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.262388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.262365 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.262572 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.262555 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.317429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.317390 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/237ddf20-334e-4c7e-8b07-6750aaa360b9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.317575 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.317544 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dnx\" (UniqueName: \"kubernetes.io/projected/237ddf20-334e-4c7e-8b07-6750aaa360b9-kube-api-access-v8dnx\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-4qpb8\" (UID: \"237ddf20-334e-4c7e-8b07-6750aaa360b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.433120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.433039 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:08.861281 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.861250 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8"] Nov 02 18:55:08.862989 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:55:08.862958 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod237ddf20_334e_4c7e_8b07_6750aaa360b9.slice/crio-f7ebe0d37691b5e89901855e0eecd42a547cc8a02348dad47f6b21442cba9416 WatchSource:0}: Error finding container f7ebe0d37691b5e89901855e0eecd42a547cc8a02348dad47f6b21442cba9416: Status 404 returned error can't find the container with id f7ebe0d37691b5e89901855e0eecd42a547cc8a02348dad47f6b21442cba9416 Nov 02 18:55:08.865348 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.865313 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:08.865438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.865393 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:08.865438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:08.865425 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:09.538712 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:09.538671 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" event={"ID":"237ddf20-334e-4c7e-8b07-6750aaa360b9","Type":"ContainerStarted","Data":"7a74f55b7ac8dfb279d40147716a18c899cffc796620bd89318e5128aeb79a33"} Nov 02 18:55:09.539261 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:09.538724 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" event={"ID":"237ddf20-334e-4c7e-8b07-6750aaa360b9","Type":"ContainerStarted","Data":"f7ebe0d37691b5e89901855e0eecd42a547cc8a02348dad47f6b21442cba9416"} Nov 02 18:55:09.759597 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:09.759537 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podStartSLOduration=1.75952329 podStartE2EDuration="1.75952329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:09.715333794 +0000 UTC m=+5496.788861000" watchObservedRunningTime="2026-11-02 18:55:09.75952329 +0000 UTC m=+5496.833050492" Nov 02 18:55:10.044380 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:10.044341 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69"] Nov 02 18:55:10.047305 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:10.047282 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a002c0-x4s69"] Nov 02 18:55:10.433691 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:10.433662 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:10.435127 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:10.435101 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:10.435264 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:10.435150 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:11.079779 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:11.079721 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff654825-580e-4095-bd64-54d77c445da0" path="/var/lib/kubelet/pods/ff654825-580e-4095-bd64-54d77c445da0/volumes" Nov 02 18:55:11.433945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:11.433906 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:11.434145 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:11.433972 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:12.434238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:12.434198 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:12.434678 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:12.434266 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:13.433660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:13.433622 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:13.433882 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:13.433693 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:14.434486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:14.434445 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:14.435057 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:14.434514 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:15.433945 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:15.433905 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:15.434147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:15.433977 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:15.518845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:15.518816 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fd65b9f5-t84fb" Nov 02 18:55:16.044187 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:16.044150 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8"] Nov 02 18:55:16.046116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:16.046096 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4edede-69cd8"] Nov 02 18:55:16.433483 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:16.433429 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" start-of-body= Nov 02 18:55:16.433718 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:16.433498 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": dial tcp 10.134.1.74:15021: connect: connection refused" Nov 02 18:55:17.080370 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:17.080327 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f752bb-62ca-48a6-95a0-0c871d543dc6" path="/var/lib/kubelet/pods/45f752bb-62ca-48a6-95a0-0c871d543dc6/volumes" Nov 02 18:55:17.438012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:17.437981 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:17.438292 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:17.438271 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:17.439017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:17.438999 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" Nov 02 18:55:19.043508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:19.043468 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p"] Nov 02 18:55:19.094041 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:19.094006 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-863dfc69-zng9p"] Nov 02 18:55:21.045661 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:21.045629 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2"] Nov 02 18:55:21.079812 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:21.079784 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a507175-a27c-406d-a42e-8bed1d13b219" path="/var/lib/kubelet/pods/1a507175-a27c-406d-a42e-8bed1d13b219/volumes" Nov 02 18:55:21.096299 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:21.096264 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a38e46a3-z6mf2"] Nov 02 18:55:23.079790 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:23.079758 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd449d2-8165-44d5-b1b6-28d774308dc9" path="/var/lib/kubelet/pods/7cd449d2-8165-44d5-b1b6-28d774308dc9/volumes" Nov 02 18:55:28.674207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:28.674170 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:55:34.044482 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:34.044401 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh"] Nov 02 18:55:34.094851 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:34.094815 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f198960e-6twjh"] Nov 02 18:55:34.954472 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:34.954430 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll"] Nov 02 18:55:34.959639 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:34.959613 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.014306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.014272 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll"] Nov 02 18:55:35.057103 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.057066 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwv78\" (UniqueName: \"kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.057509 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.057131 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.057509 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.057230 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.062933 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.062911 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:55:35.063008 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.062995 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:55:35.080104 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.080070 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937f82a5-5e88-4b89-a712-b11a67803cbc" path="/var/lib/kubelet/pods/937f82a5-5e88-4b89-a712-b11a67803cbc/volumes" Nov 02 18:55:35.158239 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.158199 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.158438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.158263 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.158438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.158361 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwv78\" (UniqueName: \"kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.159002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.158980 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.160659 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.160642 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.216660 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.216597 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwv78\" (UniqueName: \"kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78\") pod \"maas-api-revoke-sub-f904bff9-lxqll\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.272683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.272650 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:35.642745 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.642704 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll"] Nov 02 18:55:35.644454 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:55:35.644423 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0d1b42_ae4f_435f_8f6d_68e70c939a07.slice/crio-3dad1a34857f110a731910c3148adce9cdd54d591256e4637bd0787483ee199f WatchSource:0}: Error finding container 3dad1a34857f110a731910c3148adce9cdd54d591256e4637bd0787483ee199f: Status 404 returned error can't find the container with id 3dad1a34857f110a731910c3148adce9cdd54d591256e4637bd0787483ee199f Nov 02 18:55:35.691514 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:35.691480 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" event={"ID":"bf0d1b42-ae4f-435f-8f6d-68e70c939a07","Type":"ContainerStarted","Data":"3dad1a34857f110a731910c3148adce9cdd54d591256e4637bd0787483ee199f"} Nov 02 18:55:35.945980 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:55:35.945943 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:55:36.699053 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:36.698960 5647 generic.go:356] "Generic (PLEG): container finished" podID="bf0d1b42-ae4f-435f-8f6d-68e70c939a07" containerID="69df7f704a537e458d64a198f4b192957a1e4b814f22ce1f712bc9c4ec3326a2" exitCode=0 Nov 02 18:55:36.699413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:36.699047 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" event={"ID":"bf0d1b42-ae4f-435f-8f6d-68e70c939a07","Type":"ContainerDied","Data":"69df7f704a537e458d64a198f4b192957a1e4b814f22ce1f712bc9c4ec3326a2"} Nov 02 18:55:37.839371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.839344 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:37.882658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.882621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca\") pod \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " Nov 02 18:55:37.882869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.882677 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78\" (UniqueName: \"kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78\") pod \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " Nov 02 18:55:37.882869 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.882856 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca\") pod \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\" (UID: \"bf0d1b42-ae4f-435f-8f6d-68e70c939a07\") " Nov 02 18:55:37.883170 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.883148 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca" pod "bf0d1b42-ae4f-435f-8f6d-68e70c939a07" (UID: "bf0d1b42-ae4f-435f-8f6d-68e70c939a07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:55:37.883252 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.883232 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:55:37.884955 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.884930 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca" pod "bf0d1b42-ae4f-435f-8f6d-68e70c939a07" (UID: "bf0d1b42-ae4f-435f-8f6d-68e70c939a07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:55:37.885067 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.885003 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78" pod "bf0d1b42-ae4f-435f-8f6d-68e70c939a07" (UID: "bf0d1b42-ae4f-435f-8f6d-68e70c939a07"). InnerVolumeSpecName "kube-api-access-xwv78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:55:37.984886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.984805 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwv78\" (UniqueName: \"kubernetes.io/projected/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-kube-api-access-xwv78\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:55:37.984886 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:37.984834 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf0d1b42-ae4f-435f-8f6d-68e70c939a07-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:55:38.093580 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:38.093543 5647 scope.go:122] "RemoveContainer" containerID="d83ab2536310139a9ce227b641a91d57ff5654aa3ac40ea0934a3766363e608b" Nov 02 18:55:38.102825 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:38.102803 5647 scope.go:122] "RemoveContainer" containerID="58e0b847cf0b40453830aec3e941c6abdbe7b253709d03b6e31ffa5dc84b5f17" Nov 02 18:55:38.113400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:38.113378 5647 scope.go:122] "RemoveContainer" containerID="44e6acf698c9cf80e5453532771585fc29c6f3ab57a3f43f6203d74957bd13c1" Nov 02 18:55:38.710455 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:38.710421 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" Nov 02 18:55:38.710648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:38.710463 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll" event={"ID":"bf0d1b42-ae4f-435f-8f6d-68e70c939a07","Type":"ContainerDied","Data":"3dad1a34857f110a731910c3148adce9cdd54d591256e4637bd0787483ee199f"} Nov 02 18:55:38.710648 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:38.710510 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dad1a34857f110a731910c3148adce9cdd54d591256e4637bd0787483ee199f" Nov 02 18:55:47.667419 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:47.667380 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:55:53.456076 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.456040 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt"] Nov 02 18:55:53.456679 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.456657 5647 cpu_manager.go:414] "containerMap: removing container" podUID="bf0d1b42-ae4f-435f-8f6d-68e70c939a07" containerName="revoke-keys" Nov 02 18:55:53.456759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.456685 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf0d1b42-ae4f-435f-8f6d-68e70c939a07" containerName="revoke-keys" Nov 02 18:55:53.456889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.456874 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf0d1b42-ae4f-435f-8f6d-68e70c939a07" containerName="revoke-keys" Nov 02 18:55:53.460519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.460496 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.506921 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.506874 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw"] Nov 02 18:55:53.511288 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.511259 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.546598 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546555 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cps2\" (UniqueName: \"kubernetes.io/projected/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-kube-api-access-9cps2\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.546845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546642 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.546845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546683 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.546845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546703 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.546845 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546723 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.547121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546919 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.547121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.546980 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.547121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.547031 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.547121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.547085 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.562159 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.562012 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt"] Nov 02 18:55:53.632757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.631847 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw"] Nov 02 18:55:53.648602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.648565 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.649016 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.648987 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cps2\" (UniqueName: \"kubernetes.io/projected/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-kube-api-access-9cps2\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.649177 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649157 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.649311 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649294 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.649440 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649425 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.649563 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649546 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.649700 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649681 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.649882 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649863 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.650011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.649994 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.650143 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650129 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.650265 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650248 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.650393 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650376 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ckvz\" (UniqueName: \"kubernetes.io/projected/f33d9232-9eef-4f5e-9240-ff28e8d84f96-kube-api-access-8ckvz\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.650520 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650504 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.650658 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650643 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.650816 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650799 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.650954 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.650938 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.651106 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.651089 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.651238 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.651223 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.652200 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.652173 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.653602 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.653293 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.654486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.654463 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.655012 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.654959 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.657074 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.657021 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.668998 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.668913 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.673908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.673853 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.717591 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.717481 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cps2\" (UniqueName: \"kubernetes.io/projected/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-kube-api-access-9cps2\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.731002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.730919 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42e84755-3ca6-4b00-8eca-2f8c7cfeb744-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-bn7pt\" (UID: \"42e84755-3ca6-4b00-8eca-2f8c7cfeb744\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.752987 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753059 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753124 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753154 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753195 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753229 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ckvz\" (UniqueName: \"kubernetes.io/projected/f33d9232-9eef-4f5e-9240-ff28e8d84f96-kube-api-access-8ckvz\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753277 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753301 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.753505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.753349 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.755077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.754880 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.755077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.755034 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.755589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.755409 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.755589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.755552 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.756427 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.756328 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.764397 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.764342 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.765406 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.765359 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.775454 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.775428 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:53.821347 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.821258 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33d9232-9eef-4f5e-9240-ff28e8d84f96-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.822270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.822206 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ckvz\" (UniqueName: \"kubernetes.io/projected/f33d9232-9eef-4f5e-9240-ff28e8d84f96-kube-api-access-8ckvz\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-5zzpw\" (UID: \"f33d9232-9eef-4f5e-9240-ff28e8d84f96\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:53.825781 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:53.825630 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:54.243468 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.242158 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt"] Nov 02 18:55:54.244714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.244448 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:54.244714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.244532 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:54.244714 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.244578 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:54.286899 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.286870 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw"] Nov 02 18:55:54.288764 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:55:54.288701 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33d9232_9eef_4f5e_9240_ff28e8d84f96.slice/crio-000819eb83edaaa379360b5ab139a3fc909cd6df8671732c82343680bdd0f961 WatchSource:0}: Error finding container 000819eb83edaaa379360b5ab139a3fc909cd6df8671732c82343680bdd0f961: Status 404 returned error can't find the container with id 000819eb83edaaa379360b5ab139a3fc909cd6df8671732c82343680bdd0f961 Nov 02 18:55:54.291086 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.291047 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:54.291193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.291137 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:54.291193 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.291176 5647 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 02 18:55:54.817129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.817087 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" event={"ID":"42e84755-3ca6-4b00-8eca-2f8c7cfeb744","Type":"ContainerStarted","Data":"91c5e8e2b844cdcea4f51e985dbd061f03225a74654b4d1589ef2103498378af"} Nov 02 18:55:54.817931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.817904 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" event={"ID":"42e84755-3ca6-4b00-8eca-2f8c7cfeb744","Type":"ContainerStarted","Data":"fdf5736546f948facc8793c6f84471f0b366c94ce3c7047c399e3cb129be49d5"} Nov 02 18:55:54.832613 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.832573 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" event={"ID":"f33d9232-9eef-4f5e-9240-ff28e8d84f96","Type":"ContainerStarted","Data":"3ea95d2176a39fca4a607f2c3c6952cb4931b2cdaaf53dcf739d1f1cc352f4af"} Nov 02 18:55:54.832911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:54.832886 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" event={"ID":"f33d9232-9eef-4f5e-9240-ff28e8d84f96","Type":"ContainerStarted","Data":"000819eb83edaaa379360b5ab139a3fc909cd6df8671732c82343680bdd0f961"} Nov 02 18:55:55.088850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.085448 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podStartSLOduration=2.085432302 podStartE2EDuration="2.085432302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:54.957165953 +0000 UTC m=+5542.030693159" watchObservedRunningTime="2026-11-02 18:55:55.085432302 +0000 UTC m=+5542.158959505" Nov 02 18:55:55.776183 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.776142 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:55:55.777793 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.777766 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:55:55.777880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.777818 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:55:55.833771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.830418 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:55:55.833771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.830554 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:55:55.833771 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:55.830599 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:55:56.776179 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:56.776134 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:55:56.776383 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:56.776201 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:55:56.830757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:56.830459 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:55:56.830757 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:56.830533 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:55:57.254776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:57.253558 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podStartSLOduration=4.253538205 podStartE2EDuration="4.253538205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:55:55.084956799 +0000 UTC m=+5542.158484002" watchObservedRunningTime="2026-11-02 18:55:57.253538205 +0000 UTC m=+5544.327065409" Nov 02 18:55:57.254776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:57.254080 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:55:57.776907 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:57.776877 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:55:57.777171 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:57.777141 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:55:57.829286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:57.829159 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:55:57.829286 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:57.829224 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:55:58.776694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:58.776654 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:55:58.777129 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:58.776706 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:55:58.827204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:58.827163 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:55:58.827498 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:58.827471 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:55:59.776800 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:59.776764 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:55:59.777300 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:59.777269 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:55:59.832285 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:59.832244 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:55:59.832590 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:55:59.832557 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:56:00.776747 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:00.776702 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:56:00.776979 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:00.776787 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:56:00.827413 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:00.827371 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:56:00.827593 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:00.827439 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:56:01.776254 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:01.776215 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 02 18:56:01.776449 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:01.776281 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 02 18:56:01.827130 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:01.827088 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body= Nov 02 18:56:01.827523 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:01.827162 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" Nov 02 18:56:03.777314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:03.777271 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:03.778017 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:03.777981 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:56:03.847911 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:03.847856 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:03.848147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:03.847968 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:56:04.823932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:04.823869 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:04.824586 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:04.824553 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:56:04.899968 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:04.899891 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:04.900345 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:04.900277 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:56:05.869011 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:05.868957 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:05.869570 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:05.869030 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:56:05.955967 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:05.955891 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 18:56:05.956325 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:05.956265 5647 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 18:56:05.964536 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:56:05.964438 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:56:06.355917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.355880 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:56:06.356408 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.356380 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:56:06.448769 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.446286 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" Nov 02 18:56:06.560090 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.560058 5647 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:56:06.560371 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.560349 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:56:06.561046 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.561029 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" Nov 02 18:56:06.784841 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:06.784720 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:56:08.686270 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.686234 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj"] Nov 02 18:56:08.691191 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.691163 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.745625 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.745587 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj"] Nov 02 18:56:08.794100 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.794060 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:08.794293 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.794124 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:56:08.815273 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.815239 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.815465 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.815345 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhlw\" (UniqueName: \"kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.815465 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.815409 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.916225 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.916186 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.916441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.916248 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.916441 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.916301 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhlw\" (UniqueName: \"kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.916880 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.916858 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.918766 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.918727 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:08.974853 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:08.974777 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhlw\" (UniqueName: \"kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw\") pod \"maas-api-revoke-sub-f03629db-z6bkj\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:09.005568 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:09.005533 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:09.378093 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:09.378067 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj"] Nov 02 18:56:09.379986 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:56:09.379960 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18d8f20_dbd7_4e7a_9f54_424bb24965cc.slice/crio-5d038c40deb7501e2d9f68f0fbedf927082ce95e20add94a18601fcf783081eb WatchSource:0}: Error finding container 5d038c40deb7501e2d9f68f0fbedf927082ce95e20add94a18601fcf783081eb: Status 404 returned error can't find the container with id 5d038c40deb7501e2d9f68f0fbedf927082ce95e20add94a18601fcf783081eb Nov 02 18:56:09.958150 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:09.958110 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" event={"ID":"a18d8f20-dbd7-4e7a-9f54-424bb24965cc","Type":"ContainerStarted","Data":"2bd6d6f2dfb01f8fda4af32d69b41dd5b5a82167513404506b4489d56996b615"} Nov 02 18:56:09.958585 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:09.958178 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" event={"ID":"a18d8f20-dbd7-4e7a-9f54-424bb24965cc","Type":"ContainerStarted","Data":"5d038c40deb7501e2d9f68f0fbedf927082ce95e20add94a18601fcf783081eb"} Nov 02 18:56:10.072478 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:10.072362 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" podStartSLOduration=2.072340902 podStartE2EDuration="2.072340902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:10.072139327 +0000 UTC m=+5557.145666532" watchObservedRunningTime="2026-11-02 18:56:10.072340902 +0000 UTC m=+5557.145868107" Nov 02 18:56:10.966184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:10.966147 5647 generic.go:356] "Generic (PLEG): container finished" podID="a18d8f20-dbd7-4e7a-9f54-424bb24965cc" containerID="2bd6d6f2dfb01f8fda4af32d69b41dd5b5a82167513404506b4489d56996b615" exitCode=0 Nov 02 18:56:10.966184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:10.966188 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" event={"ID":"a18d8f20-dbd7-4e7a-9f54-424bb24965cc","Type":"ContainerDied","Data":"2bd6d6f2dfb01f8fda4af32d69b41dd5b5a82167513404506b4489d56996b615"} Nov 02 18:56:12.109426 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.109395 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:12.150765 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.150703 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw\" (UniqueName: \"kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw\") pod \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " Nov 02 18:56:12.150918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.150799 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca\") pod \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " Nov 02 18:56:12.150918 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.150893 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca\") pod \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\" (UID: \"a18d8f20-dbd7-4e7a-9f54-424bb24965cc\") " Nov 02 18:56:12.151152 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.151121 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca" pod "a18d8f20-dbd7-4e7a-9f54-424bb24965cc" (UID: "a18d8f20-dbd7-4e7a-9f54-424bb24965cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:12.151335 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.151318 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:12.153120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.153085 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw" pod "a18d8f20-dbd7-4e7a-9f54-424bb24965cc" (UID: "a18d8f20-dbd7-4e7a-9f54-424bb24965cc"). InnerVolumeSpecName "kube-api-access-gqhlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:12.153120 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.153099 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca" pod "a18d8f20-dbd7-4e7a-9f54-424bb24965cc" (UID: "a18d8f20-dbd7-4e7a-9f54-424bb24965cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:12.252066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.251983 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqhlw\" (UniqueName: \"kubernetes.io/projected/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-kube-api-access-gqhlw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:12.252066 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.252013 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18d8f20-dbd7-4e7a-9f54-424bb24965cc-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:12.986269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.986232 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" event={"ID":"a18d8f20-dbd7-4e7a-9f54-424bb24965cc","Type":"ContainerDied","Data":"5d038c40deb7501e2d9f68f0fbedf927082ce95e20add94a18601fcf783081eb"} Nov 02 18:56:12.986269 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.986259 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 18:56:12.986508 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:12.986304 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d038c40deb7501e2d9f68f0fbedf927082ce95e20add94a18601fcf783081eb" Nov 02 18:56:15.170184 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:15.170144 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:56:28.233573 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.233537 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk"] Nov 02 18:56:28.234181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.234044 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a18d8f20-dbd7-4e7a-9f54-424bb24965cc" containerName="revoke-keys" Nov 02 18:56:28.234181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.234060 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18d8f20-dbd7-4e7a-9f54-424bb24965cc" containerName="revoke-keys" Nov 02 18:56:28.234181 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.234136 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18d8f20-dbd7-4e7a-9f54-424bb24965cc" containerName="revoke-keys" Nov 02 18:56:28.237505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.237488 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.285545 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.285502 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmlw\" (UniqueName: \"kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.285723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.285561 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.285723 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.285656 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.293486 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.293453 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk"] Nov 02 18:56:28.341776 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.341711 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:28.341959 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.341826 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:56:28.387051 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.387013 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmlw\" (UniqueName: \"kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.387256 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.387058 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.387256 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.387104 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.387832 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.387806 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.389775 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.389744 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.444477 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.444444 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmlw\" (UniqueName: \"kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw\") pod \"maas-api-revoke-sub-0165d7d2-hhlbk\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.553527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.553444 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:28.925309 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:28.925285 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk"] Nov 02 18:56:28.928014 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:56:28.927985 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25a7447_026e_401e_a8be_4f0c4bcfed66.slice/crio-88d90a1bc17fb045a96314605dc1ef1d9c51d217417ac8c1675b81b213a2079d WatchSource:0}: Error finding container 88d90a1bc17fb045a96314605dc1ef1d9c51d217417ac8c1675b81b213a2079d: Status 404 returned error can't find the container with id 88d90a1bc17fb045a96314605dc1ef1d9c51d217417ac8c1675b81b213a2079d Nov 02 18:56:29.107532 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:29.107498 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" event={"ID":"a25a7447-026e-401e-a8be-4f0c4bcfed66","Type":"ContainerStarted","Data":"88d90a1bc17fb045a96314605dc1ef1d9c51d217417ac8c1675b81b213a2079d"} Nov 02 18:56:30.114881 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:30.114845 5647 generic.go:356] "Generic (PLEG): container finished" podID="a25a7447-026e-401e-a8be-4f0c4bcfed66" containerID="b3def1b250102277816d02c565195332d0175cf9c7d70730767c502ce95d891e" exitCode=0 Nov 02 18:56:30.115308 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:30.114929 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" event={"ID":"a25a7447-026e-401e-a8be-4f0c4bcfed66","Type":"ContainerDied","Data":"b3def1b250102277816d02c565195332d0175cf9c7d70730767c502ce95d891e"} Nov 02 18:56:31.256932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.256907 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:31.316906 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.316878 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca\") pod \"a25a7447-026e-401e-a8be-4f0c4bcfed66\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " Nov 02 18:56:31.317085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.316966 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw\" (UniqueName: \"kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw\") pod \"a25a7447-026e-401e-a8be-4f0c4bcfed66\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " Nov 02 18:56:31.317085 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.316996 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca\") pod \"a25a7447-026e-401e-a8be-4f0c4bcfed66\" (UID: \"a25a7447-026e-401e-a8be-4f0c4bcfed66\") " Nov 02 18:56:31.317391 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.317367 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca" pod "a25a7447-026e-401e-a8be-4f0c4bcfed66" (UID: "a25a7447-026e-401e-a8be-4f0c4bcfed66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:31.319034 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.319013 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca" pod "a25a7447-026e-401e-a8be-4f0c4bcfed66" (UID: "a25a7447-026e-401e-a8be-4f0c4bcfed66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:31.319174 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.319155 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw" pod "a25a7447-026e-401e-a8be-4f0c4bcfed66" (UID: "a25a7447-026e-401e-a8be-4f0c4bcfed66"). InnerVolumeSpecName "kube-api-access-hlmlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:31.417864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.417788 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:31.417864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.417817 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmlw\" (UniqueName: \"kubernetes.io/projected/a25a7447-026e-401e-a8be-4f0c4bcfed66-kube-api-access-hlmlw\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:31.417864 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:31.417827 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a25a7447-026e-401e-a8be-4f0c4bcfed66-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:32.126357 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:32.126321 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" event={"ID":"a25a7447-026e-401e-a8be-4f0c4bcfed66","Type":"ContainerDied","Data":"88d90a1bc17fb045a96314605dc1ef1d9c51d217417ac8c1675b81b213a2079d"} Nov 02 18:56:32.126589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:32.126371 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d90a1bc17fb045a96314605dc1ef1d9c51d217417ac8c1675b81b213a2079d" Nov 02 18:56:32.126589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:32.126337 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk" Nov 02 18:56:35.948827 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:56:35.948784 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:56:38.147147 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.147108 5647 scope.go:122] "RemoveContainer" containerID="8eb225324d41c83680bcd7d47960104e069db11e11005f932b5ad9853f0c3240" Nov 02 18:56:38.158795 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.158768 5647 scope.go:122] "RemoveContainer" containerID="5454bbeb9b03a7104142cb4e8c523fb38214634cf4dc46e944ebb77778913e08" Nov 02 18:56:38.168355 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.168338 5647 scope.go:122] "RemoveContainer" containerID="0600e9d5975f8f289c140d0d0272386d97ed592fbee55a22054f1f8e6c911957" Nov 02 18:56:38.176314 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.176293 5647 scope.go:122] "RemoveContainer" containerID="7347a37de8774ed59e742ffb8e4b51a96058fcccb525e729903ccb20cd1ad452" Nov 02 18:56:38.184102 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.184079 5647 scope.go:122] "RemoveContainer" containerID="d596dec56fe761047ee8219538d96e309ffd052898c72804c6bb95549bedc444" Nov 02 18:56:38.191986 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.191968 5647 scope.go:122] "RemoveContainer" containerID="e206176227a05b32c2b6bdad412273abae57e9e9e2ad2a755c2ac6372bb4a44c" Nov 02 18:56:38.199564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:38.199549 5647 scope.go:122] "RemoveContainer" containerID="e2aeef64a10dd18aa8e79d8333b8180d66b8cd87ed83d01df2a98d9e38f251ad" Nov 02 18:56:40.157169 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.157137 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth"] Nov 02 18:56:40.157632 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.157617 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a25a7447-026e-401e-a8be-4f0c4bcfed66" containerName="revoke-keys" Nov 02 18:56:40.157683 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.157634 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25a7447-026e-401e-a8be-4f0c4bcfed66" containerName="revoke-keys" Nov 02 18:56:40.157759 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.157749 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25a7447-026e-401e-a8be-4f0c4bcfed66" containerName="revoke-keys" Nov 02 18:56:40.162609 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.162590 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.210204 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.210165 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.210385 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.210308 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8qc\" (UniqueName: \"kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.210385 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.210353 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.218400 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.218373 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth"] Nov 02 18:56:40.266931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.266893 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:56:40.266931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.266927 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:56:40.311778 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.311714 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8qc\" (UniqueName: \"kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.311975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.311813 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.311975 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.311890 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.312633 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.312567 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.314430 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.314407 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.371850 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.371808 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8qc\" (UniqueName: \"kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc\") pod \"maas-api-revoke-sub-6b19338f-64fth\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.476564 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.476474 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:40.851060 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:40.851027 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth"] Nov 02 18:56:40.854435 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:56:40.854404 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c224c07_a2fd_48e7_971c_10d6b5e359e5.slice/crio-036db2b82fff7627e24a87fef4dd457198946f6e3b730f2bfdb8c4ed3ff662f2 WatchSource:0}: Error finding container 036db2b82fff7627e24a87fef4dd457198946f6e3b730f2bfdb8c4ed3ff662f2: Status 404 returned error can't find the container with id 036db2b82fff7627e24a87fef4dd457198946f6e3b730f2bfdb8c4ed3ff662f2 Nov 02 18:56:41.182319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:41.182246 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" event={"ID":"9c224c07-a2fd-48e7-971c-10d6b5e359e5","Type":"ContainerStarted","Data":"370a754c355ac1f7c35f7f70c215f847f516119509892657a2da60a6cc34eae0"} Nov 02 18:56:41.182319 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:41.182302 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" event={"ID":"9c224c07-a2fd-48e7-971c-10d6b5e359e5","Type":"ContainerStarted","Data":"036db2b82fff7627e24a87fef4dd457198946f6e3b730f2bfdb8c4ed3ff662f2"} Nov 02 18:56:41.300002 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:41.299905 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" podStartSLOduration=1.29989083 podStartE2EDuration="1.29989083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:41.298319652 +0000 UTC m=+5588.371846855" watchObservedRunningTime="2026-11-02 18:56:41.29989083 +0000 UTC m=+5588.373418032" Nov 02 18:56:42.189268 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:42.189235 5647 generic.go:356] "Generic (PLEG): container finished" podID="9c224c07-a2fd-48e7-971c-10d6b5e359e5" containerID="370a754c355ac1f7c35f7f70c215f847f516119509892657a2da60a6cc34eae0" exitCode=0 Nov 02 18:56:42.189693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:42.189325 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" event={"ID":"9c224c07-a2fd-48e7-971c-10d6b5e359e5","Type":"ContainerDied","Data":"370a754c355ac1f7c35f7f70c215f847f516119509892657a2da60a6cc34eae0"} Nov 02 18:56:43.328257 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.328231 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:43.445656 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.445621 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc\" (UniqueName: \"kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc\") pod \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " Nov 02 18:56:43.445889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.445751 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca\") pod \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " Nov 02 18:56:43.445889 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.445817 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca\") pod \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\" (UID: \"9c224c07-a2fd-48e7-971c-10d6b5e359e5\") " Nov 02 18:56:43.446321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.446126 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca" pod "9c224c07-a2fd-48e7-971c-10d6b5e359e5" (UID: "9c224c07-a2fd-48e7-971c-10d6b5e359e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:56:43.446321 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.446246 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:43.448245 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.448220 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc" pod "9c224c07-a2fd-48e7-971c-10d6b5e359e5" (UID: "9c224c07-a2fd-48e7-971c-10d6b5e359e5"). InnerVolumeSpecName "kube-api-access-9t8qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:56:43.448425 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.448274 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca" pod "9c224c07-a2fd-48e7-971c-10d6b5e359e5" (UID: "9c224c07-a2fd-48e7-971c-10d6b5e359e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:56:43.547710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.547676 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8qc\" (UniqueName: \"kubernetes.io/projected/9c224c07-a2fd-48e7-971c-10d6b5e359e5-kube-api-access-9t8qc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:43.547710 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:43.547706 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c224c07-a2fd-48e7-971c-10d6b5e359e5-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:56:44.202429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:44.202394 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" Nov 02 18:56:44.202429 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:44.202409 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth" event={"ID":"9c224c07-a2fd-48e7-971c-10d6b5e359e5","Type":"ContainerDied","Data":"036db2b82fff7627e24a87fef4dd457198946f6e3b730f2bfdb8c4ed3ff662f2"} Nov 02 18:56:44.202676 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:44.202459 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036db2b82fff7627e24a87fef4dd457198946f6e3b730f2bfdb8c4ed3ff662f2" Nov 02 18:56:52.959388 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:52.959350 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:56:55.737116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.737076 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24"] Nov 02 18:56:55.737836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.737811 5647 cpu_manager.go:414] "containerMap: removing container" podUID="9c224c07-a2fd-48e7-971c-10d6b5e359e5" containerName="revoke-keys" Nov 02 18:56:55.737836 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.737837 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c224c07-a2fd-48e7-971c-10d6b5e359e5" containerName="revoke-keys" Nov 02 18:56:55.738048 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.737975 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c224c07-a2fd-48e7-971c-10d6b5e359e5" containerName="revoke-keys" Nov 02 18:56:55.741760 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.741718 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:55.796165 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.796128 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24"] Nov 02 18:56:55.835653 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.835613 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs\") pod \"e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:55.836121 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.836080 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp5w\" (UniqueName: \"kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w\") pod \"e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:55.844511 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.844473 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4397aa02-epp-sa-dockercfg-dfm6z\"" Nov 02 18:56:55.844929 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.844894 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4397aa02-kserve-self-signed-certs\"" Nov 02 18:56:55.943211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.943173 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs\") pod \"e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:55.943485 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.943459 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp5w\" (UniqueName: \"kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w\") pod \"e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:55.967366 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:55.964318 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs\") pod \"e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:56.021500 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:56.021409 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp5w\" (UniqueName: \"kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w\") pod \"e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:56.066696 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:56.066374 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:56.481275 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:56.481252 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24"] Nov 02 18:56:56.483247 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:56:56.483221 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31045f9_10a7_405f_a0ba_6c827c281ab8.slice/crio-5945a16ee82ba7af4488f5babad08dd70c51b736804a86639643695a3f145ab3 WatchSource:0}: Error finding container 5945a16ee82ba7af4488f5babad08dd70c51b736804a86639643695a3f145ab3: Status 404 returned error can't find the container with id 5945a16ee82ba7af4488f5babad08dd70c51b736804a86639643695a3f145ab3 Nov 02 18:56:57.343390 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:57.343350 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" event={"ID":"a31045f9-10a7-405f-a0ba-6c827c281ab8","Type":"ContainerStarted","Data":"5945a16ee82ba7af4488f5babad08dd70c51b736804a86639643695a3f145ab3"} Nov 02 18:56:59.354694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:59.354654 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" event={"ID":"a31045f9-10a7-405f-a0ba-6c827c281ab8","Type":"ContainerStarted","Data":"b495e6a97835411d93723ca083f6991fe512704b7406ed3e73313bd2f7f24ec2"} Nov 02 18:56:59.355207 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:59.354795 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:56:59.477932 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:56:59.477856 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" podStartSLOduration=2.350272259 podStartE2EDuration="4.477835937s" totalImagesPullingTime="2.127563678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:56:59.469559906 +0000 UTC m=+5606.543087135" watchObservedRunningTime="2026-11-02 18:56:59.477835937 +0000 UTC m=+5606.551363141" Nov 02 18:57:05.949283 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:57:05.949249 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:57:06.386505 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:06.386471 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:57:25.044858 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:25.044819 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c"] Nov 02 18:57:25.095295 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:25.095261 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78dfeaf0-zht6c"] Nov 02 18:57:27.080603 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:27.080570 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b1ef68-41e2-42a9-b4ad-4d70feae21a7" path="/var/lib/kubelet/pods/19b1ef68-41e2-42a9-b4ad-4d70feae21a7/volumes" Nov 02 18:57:30.365280 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:30.365249 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:57:35.948262 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:57:35.948212 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:57:37.052833 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:37.052786 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf"] Nov 02 18:57:37.103199 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:37.103170 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-048cb7fb-s2zbf"] Nov 02 18:57:39.080209 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:39.080172 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d" path="/var/lib/kubelet/pods/893f03c7-a0b1-4e81-a3a8-9d6fdc5c8b0d/volumes" Nov 02 18:57:39.238584 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:57:39.238551 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:58:03.749944 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:03.749905 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:58:03.834671 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:03.834638 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24"] Nov 02 18:58:05.947799 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:58:05.947728 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:58:10.379926 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:10.379864 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:58:14.962589 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:14.962503 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:58:18.856303 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:18.856255 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" containerID="cri-o://b495e6a97835411d93723ca083f6991fe512704b7406ed3e73313bd2f7f24ec2" gracePeriod=45 Nov 02 18:58:20.364797 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:20.364745 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:58:30.365547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:30.365509 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:58:30.365985 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:30.365627 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:58:35.011306 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:35.011273 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:58:35.016131 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:35.016107 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 18:58:35.947977 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:58:35.947938 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:58:38.251018 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:38.250978 5647 scope.go:122] "RemoveContainer" containerID="e5ce56078d502f8bc9536073e8477ce0c17fcffb13752994100f2c7dedb28d21" Nov 02 18:58:38.261116 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:38.261082 5647 scope.go:122] "RemoveContainer" containerID="aefb4b15b9bd0609485ca6dc896aa2e85b9286c478586eb8e829c84d845212a3" Nov 02 18:58:40.365495 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:40.365454 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 18:58:41.247315 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:41.247281 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:58:49.072917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.072877 5647 generic.go:356] "Generic (PLEG): container finished" podID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerID="b495e6a97835411d93723ca083f6991fe512704b7406ed3e73313bd2f7f24ec2" exitCode=0 Nov 02 18:58:49.072917 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.072902 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" event={"ID":"a31045f9-10a7-405f-a0ba-6c827c281ab8","Type":"ContainerDied","Data":"b495e6a97835411d93723ca083f6991fe512704b7406ed3e73313bd2f7f24ec2"} Nov 02 18:58:49.119694 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.119624 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:58:49.222840 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.222805 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs\" (UniqueName: \"kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs\") pod \"a31045f9-10a7-405f-a0ba-6c827c281ab8\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " Nov 02 18:58:49.223043 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.222939 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w\" (UniqueName: \"kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w\") pod \"a31045f9-10a7-405f-a0ba-6c827c281ab8\" (UID: \"a31045f9-10a7-405f-a0ba-6c827c281ab8\") " Nov 02 18:58:49.225396 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.225361 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs" pod "a31045f9-10a7-405f-a0ba-6c827c281ab8" (UID: "a31045f9-10a7-405f-a0ba-6c827c281ab8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:58:49.225535 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.225462 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w" pod "a31045f9-10a7-405f-a0ba-6c827c281ab8" (UID: "a31045f9-10a7-405f-a0ba-6c827c281ab8"). InnerVolumeSpecName "kube-api-access-qzp5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:58:49.324324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.324280 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzp5w\" (UniqueName: \"kubernetes.io/projected/a31045f9-10a7-405f-a0ba-6c827c281ab8-kube-api-access-qzp5w\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:58:49.324324 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:49.324317 5647 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a31045f9-10a7-405f-a0ba-6c827c281ab8-tls-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:58:50.079527 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:50.079493 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" event={"ID":"a31045f9-10a7-405f-a0ba-6c827c281ab8","Type":"ContainerDied","Data":"5945a16ee82ba7af4488f5babad08dd70c51b736804a86639643695a3f145ab3"} Nov 02 18:58:50.080077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:50.079560 5647 scope.go:122] "RemoveContainer" containerID="b495e6a97835411d93723ca083f6991fe512704b7406ed3e73313bd2f7f24ec2" Nov 02 18:58:50.080077 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:50.079563 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24" Nov 02 18:58:50.245693 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:50.245655 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24"] Nov 02 18:58:50.297211 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:50.297178 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4397aa02-kserve-router-scheduler-ffb58f9c-qkv24"] Nov 02 18:58:51.080808 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:51.080772 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" path="/var/lib/kubelet/pods/a31045f9-10a7-405f-a0ba-6c827c281ab8/volumes" Nov 02 18:58:59.960950 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:58:59.960910 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:59:05.948375 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:59:05.948332 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:59:25.355381 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:25.355346 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 18:59:35.948482 ip-10-0-128-71 kubenswrapper[5647]: E1102 18:59:35.948440 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 18:59:40.405952 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.405912 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn"] Nov 02 18:59:40.406405 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.406389 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" Nov 02 18:59:40.406452 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.406409 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" Nov 02 18:59:40.406547 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.406537 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31045f9-10a7-405f-a0ba-6c827c281ab8" containerName="main" Nov 02 18:59:40.410938 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.410920 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.467783 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.467728 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn"] Nov 02 18:59:40.491519 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.491480 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 18:59:40.491719 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.491582 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 18:59:40.581644 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.581608 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8b5n\" (UniqueName: \"kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.581891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.581683 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.581891 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.581815 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.682908 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.682813 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.683087 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.682930 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8b5n\" (UniqueName: \"kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.683087 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.683020 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.683667 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.683640 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.685480 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.685457 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:40.741136 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:40.741097 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8b5n\" (UniqueName: \"kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n\") pod \"maas-api-revoke-sub-e7f3bd92-g4rvn\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:41.033857 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:41.033773 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:41.411080 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:41.411054 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn"] Nov 02 18:59:41.413254 ip-10-0-128-71 kubenswrapper[5647]: W1102 18:59:41.413220 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3546b94b_9c02_45d0_becc_64034f027be3.slice/crio-c9a0d5cf6a1a16a375018caf3974e19d48ed5b285f86cd0632a93792739f6868 WatchSource:0}: Error finding container c9a0d5cf6a1a16a375018caf3974e19d48ed5b285f86cd0632a93792739f6868: Status 404 returned error can't find the container with id c9a0d5cf6a1a16a375018caf3974e19d48ed5b285f86cd0632a93792739f6868 Nov 02 18:59:42.351992 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:42.351952 5647 generic.go:356] "Generic (PLEG): container finished" podID="3546b94b-9c02-45d0-becc-64034f027be3" containerID="774b0060f1533739461d669d6b216a186fe63d943d1978445ab8cfc1e8916619" exitCode=0 Nov 02 18:59:42.352195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:42.352038 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" event={"ID":"3546b94b-9c02-45d0-becc-64034f027be3","Type":"ContainerDied","Data":"774b0060f1533739461d669d6b216a186fe63d943d1978445ab8cfc1e8916619"} Nov 02 18:59:42.352195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:42.352089 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" event={"ID":"3546b94b-9c02-45d0-becc-64034f027be3","Type":"ContainerStarted","Data":"c9a0d5cf6a1a16a375018caf3974e19d48ed5b285f86cd0632a93792739f6868"} Nov 02 18:59:43.497688 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.497661 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:43.504931 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.504895 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n\" (UniqueName: \"kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n\") pod \"3546b94b-9c02-45d0-becc-64034f027be3\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " Nov 02 18:59:43.505097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.504940 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca\") pod \"3546b94b-9c02-45d0-becc-64034f027be3\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " Nov 02 18:59:43.505097 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.505020 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca\") pod \"3546b94b-9c02-45d0-becc-64034f027be3\" (UID: \"3546b94b-9c02-45d0-becc-64034f027be3\") " Nov 02 18:59:43.505438 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.505409 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca" pod "3546b94b-9c02-45d0-becc-64034f027be3" (UID: "3546b94b-9c02-45d0-becc-64034f027be3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 18:59:43.507195 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.507169 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n" pod "3546b94b-9c02-45d0-becc-64034f027be3" (UID: "3546b94b-9c02-45d0-becc-64034f027be3"). InnerVolumeSpecName "kube-api-access-w8b5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 18:59:43.507307 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.507270 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca" pod "3546b94b-9c02-45d0-becc-64034f027be3" (UID: "3546b94b-9c02-45d0-becc-64034f027be3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 18:59:43.606079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.606044 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:59:43.606079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.606072 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8b5n\" (UniqueName: \"kubernetes.io/projected/3546b94b-9c02-45d0-becc-64034f027be3-kube-api-access-w8b5n\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:59:43.606079 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:43.606082 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3546b94b-9c02-45d0-becc-64034f027be3-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 18:59:44.366923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:44.366888 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" Nov 02 18:59:44.366923 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:44.366914 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn" event={"ID":"3546b94b-9c02-45d0-becc-64034f027be3","Type":"ContainerDied","Data":"c9a0d5cf6a1a16a375018caf3974e19d48ed5b285f86cd0632a93792739f6868"} Nov 02 18:59:44.367218 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:44.366968 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a0d5cf6a1a16a375018caf3974e19d48ed5b285f86cd0632a93792739f6868" Nov 02 18:59:50.053474 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:50.053423 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj"] Nov 02 18:59:50.102861 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:50.102826 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-4rdtj"] Nov 02 18:59:51.080501 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:51.080463 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd" path="/var/lib/kubelet/pods/7fa6939d-13a0-4ceb-bc3d-4f7f60ac55fd/volumes" Nov 02 18:59:53.376244 ip-10-0-128-71 kubenswrapper[5647]: I1102 18:59:53.376205 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:00:00.151692 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.151644 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd"] Nov 02 19:00:00.152354 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.152332 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3546b94b-9c02-45d0-becc-64034f027be3" containerName="revoke-keys" Nov 02 19:00:00.152446 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.152357 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3546b94b-9c02-45d0-becc-64034f027be3" containerName="revoke-keys" Nov 02 19:00:00.152507 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.152478 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3546b94b-9c02-45d0-becc-64034f027be3" containerName="revoke-keys" Nov 02 19:00:00.155683 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.155662 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.210999 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.210966 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd"] Nov 02 19:00:00.259433 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.259403 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:00:00.278719 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.278683 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8pkc\" (UniqueName: \"kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc\") pod \"maas-api-key-cleanup-29894100-wbrzd\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.278964 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.278764 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894100-wbrzd\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.379830 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.379791 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8pkc\" (UniqueName: \"kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc\") pod \"maas-api-key-cleanup-29894100-wbrzd\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.380004 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.379850 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894100-wbrzd\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.380541 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.380520 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894100-wbrzd\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.437668 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.437586 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8pkc\" (UniqueName: \"kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc\") pod \"maas-api-key-cleanup-29894100-wbrzd\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.468913 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.468875 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:00.840598 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:00.840553 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd"] Nov 02 19:00:00.843022 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:00:00.842994 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5739c2_992f_4067_aac2_971c17c3d676.slice/crio-26dd2809f2b442cc1f6b730848e0c8e147d33d9ba7128d69f712efebfee0dd4d WatchSource:0}: Error finding container 26dd2809f2b442cc1f6b730848e0c8e147d33d9ba7128d69f712efebfee0dd4d: Status 404 returned error can't find the container with id 26dd2809f2b442cc1f6b730848e0c8e147d33d9ba7128d69f712efebfee0dd4d Nov 02 19:00:01.453891 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:01.453800 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerStarted","Data":"7f16fc5dd1f4c0d73985d7683feb25c28bb1efc96e326271c5a2c8ba43577081"} Nov 02 19:00:01.453891 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:01.453860 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerStarted","Data":"26dd2809f2b442cc1f6b730848e0c8e147d33d9ba7128d69f712efebfee0dd4d"} Nov 02 19:00:01.569665 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:01.569610 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" podStartSLOduration=1.569595962 podStartE2EDuration="1.569595962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:00:01.56752215 +0000 UTC m=+5788.641049363" watchObservedRunningTime="2026-11-02 19:00:01.569595962 +0000 UTC m=+5788.643123165" Nov 02 19:00:02.461193 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:02.461159 5647 generic.go:356] "Generic (PLEG): container finished" podID="4e5739c2-992f-4067-aac2-971c17c3d676" containerID="7f16fc5dd1f4c0d73985d7683feb25c28bb1efc96e326271c5a2c8ba43577081" exitCode=60 Nov 02 19:00:02.461631 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:02.461216 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerDied","Data":"7f16fc5dd1f4c0d73985d7683feb25c28bb1efc96e326271c5a2c8ba43577081"} Nov 02 19:00:02.461631 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:02.461481 5647 scope.go:122] "RemoveContainer" containerID="7f16fc5dd1f4c0d73985d7683feb25c28bb1efc96e326271c5a2c8ba43577081" Nov 02 19:00:03.338327 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:03.338288 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:00:03.468018 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:03.467926 5647 generic.go:356] "Generic (PLEG): container finished" podID="4e5739c2-992f-4067-aac2-971c17c3d676" containerID="442523d4ec15e26b2c82c1d468abbfe96fa37cc7649c697bc8b62ab8a335e950" exitCode=60 Nov 02 19:00:03.468469 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:03.468019 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerDied","Data":"442523d4ec15e26b2c82c1d468abbfe96fa37cc7649c697bc8b62ab8a335e950"} Nov 02 19:00:03.468469 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:03.468086 5647 scope.go:122] "RemoveContainer" containerID="7f16fc5dd1f4c0d73985d7683feb25c28bb1efc96e326271c5a2c8ba43577081" Nov 02 19:00:03.468469 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:03.468325 5647 scope.go:122] "RemoveContainer" containerID="442523d4ec15e26b2c82c1d468abbfe96fa37cc7649c697bc8b62ab8a335e950" Nov 02 19:00:03.468663 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:00:03.468575 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894100-wbrzd_odh-ai-gateway-infra(4e5739c2-992f-4067-aac2-971c17c3d676)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" Nov 02 19:00:04.475256 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:04.475226 5647 scope.go:122] "RemoveContainer" containerID="442523d4ec15e26b2c82c1d468abbfe96fa37cc7649c697bc8b62ab8a335e950" Nov 02 19:00:04.475795 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:00:04.475472 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894100-wbrzd_odh-ai-gateway-infra(4e5739c2-992f-4067-aac2-971c17c3d676)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" Nov 02 19:00:05.947366 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:00:05.947321 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:00:14.073717 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:14.073676 5647 scope.go:122] "RemoveContainer" containerID="442523d4ec15e26b2c82c1d468abbfe96fa37cc7649c697bc8b62ab8a335e950" Nov 02 19:00:14.527983 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:14.527945 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerStarted","Data":"eeeedf43b40ddd68d8b121a75dcab089be89fa901e99cdc11aa22f3f53d7ef97"} Nov 02 19:00:15.534800 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:15.534766 5647 generic.go:356] "Generic (PLEG): container finished" podID="4e5739c2-992f-4067-aac2-971c17c3d676" containerID="eeeedf43b40ddd68d8b121a75dcab089be89fa901e99cdc11aa22f3f53d7ef97" exitCode=60 Nov 02 19:00:15.535280 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:15.534838 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerDied","Data":"eeeedf43b40ddd68d8b121a75dcab089be89fa901e99cdc11aa22f3f53d7ef97"} Nov 02 19:00:15.535280 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:15.534901 5647 scope.go:122] "RemoveContainer" containerID="442523d4ec15e26b2c82c1d468abbfe96fa37cc7649c697bc8b62ab8a335e950" Nov 02 19:00:15.535280 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:15.535206 5647 scope.go:122] "RemoveContainer" containerID="eeeedf43b40ddd68d8b121a75dcab089be89fa901e99cdc11aa22f3f53d7ef97" Nov 02 19:00:15.535494 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:00:15.535471 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894100-wbrzd_odh-ai-gateway-infra(4e5739c2-992f-4067-aac2-971c17c3d676)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" Nov 02 19:00:15.659361 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:15.659329 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd"] Nov 02 19:00:16.684619 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.684586 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:16.862781 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.862746 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca\") pod \"4e5739c2-992f-4067-aac2-971c17c3d676\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " Nov 02 19:00:16.863014 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.862883 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc\" (UniqueName: \"kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc\") pod \"4e5739c2-992f-4067-aac2-971c17c3d676\" (UID: \"4e5739c2-992f-4067-aac2-971c17c3d676\") " Nov 02 19:00:16.863145 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.863122 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca" pod "4e5739c2-992f-4067-aac2-971c17c3d676" (UID: "4e5739c2-992f-4067-aac2-971c17c3d676"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:00:16.863273 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.863256 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5739c2-992f-4067-aac2-971c17c3d676-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:00:16.865114 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.865092 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc" pod "4e5739c2-992f-4067-aac2-971c17c3d676" (UID: "4e5739c2-992f-4067-aac2-971c17c3d676"). InnerVolumeSpecName "kube-api-access-x8pkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:00:16.963995 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:16.963958 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8pkc\" (UniqueName: \"kubernetes.io/projected/4e5739c2-992f-4067-aac2-971c17c3d676-kube-api-access-x8pkc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:00:17.548569 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:17.548536 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" Nov 02 19:00:17.548825 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:17.548579 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd" event={"ID":"4e5739c2-992f-4067-aac2-971c17c3d676","Type":"ContainerDied","Data":"26dd2809f2b442cc1f6b730848e0c8e147d33d9ba7128d69f712efebfee0dd4d"} Nov 02 19:00:17.548825 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:17.548645 5647 scope.go:122] "RemoveContainer" containerID="eeeedf43b40ddd68d8b121a75dcab089be89fa901e99cdc11aa22f3f53d7ef97" Nov 02 19:00:17.714834 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:17.714799 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd"] Nov 02 19:00:17.765266 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:17.765207 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894100-wbrzd"] Nov 02 19:00:19.079459 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:19.079420 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" path="/var/lib/kubelet/pods/4e5739c2-992f-4067-aac2-971c17c3d676/volumes" Nov 02 19:00:35.949843 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:00:35.949789 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:00:38.044176 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:38.044134 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll"] Nov 02 19:00:38.094853 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:38.094816 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-lxqll"] Nov 02 19:00:38.298825 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:38.298705 5647 scope.go:122] "RemoveContainer" containerID="196b52d1c4fb46114f81cc196648d925017ce90ddf4db71ec88716b5d6563035" Nov 02 19:00:39.080069 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:39.080025 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf0d1b42-ae4f-435f-8f6d-68e70c939a07" path="/var/lib/kubelet/pods/bf0d1b42-ae4f-435f-8f6d-68e70c939a07/volumes" Nov 02 19:00:58.058218 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:00:58.058176 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:01:05.947240 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:01:05.947192 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:01:13.044108 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:13.044023 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj"] Nov 02 19:01:13.094525 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:13.094490 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj"] Nov 02 19:01:13.127582 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:01:13.127542 5647 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-f03629db-z6bkj\" is forbidden: User \"system:node:ip-10-0-128-71.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-71.us-west-1.compute.internal' and this object" podUID="a18d8f20-dbd7-4e7a-9f54-424bb24965cc" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-z6bkj" Nov 02 19:01:15.080257 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:15.080214 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18d8f20-dbd7-4e7a-9f54-424bb24965cc" path="/var/lib/kubelet/pods/a18d8f20-dbd7-4e7a-9f54-424bb24965cc/volumes" Nov 02 19:01:25.790125 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:25.790090 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:01:32.044564 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:32.044526 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk"] Nov 02 19:01:32.095388 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:32.095362 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hhlbk"] Nov 02 19:01:33.079493 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:33.079448 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25a7447-026e-401e-a8be-4f0c4bcfed66" path="/var/lib/kubelet/pods/a25a7447-026e-401e-a8be-4f0c4bcfed66/volumes" Nov 02 19:01:35.946930 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:01:35.946890 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:01:38.325151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:38.325116 5647 scope.go:122] "RemoveContainer" containerID="8850ce0955f59d21a0c9fbca357db98c9a563328080bbafad1bad156151c5451" Nov 02 19:01:38.333854 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:38.333830 5647 scope.go:122] "RemoveContainer" containerID="69df7f704a537e458d64a198f4b192957a1e4b814f22ce1f712bc9c4ec3326a2" Nov 02 19:01:44.045933 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:44.045890 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth"] Nov 02 19:01:44.096321 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:44.096286 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-64fth"] Nov 02 19:01:45.079673 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:45.079628 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c224c07-a2fd-48e7-971c-10d6b5e359e5" path="/var/lib/kubelet/pods/9c224c07-a2fd-48e7-971c-10d6b5e359e5/volumes" Nov 02 19:01:50.736566 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.736529 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p"] Nov 02 19:01:50.737036 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.736954 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:01:50.737036 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.736969 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:01:50.737036 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.736991 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:01:50.737036 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.736996 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:01:50.737186 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.737080 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:01:50.737186 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.737095 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:01:50.740127 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.740109 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.796221 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.796177 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p"] Nov 02 19:01:50.844881 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.844843 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:01:50.845054 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.844919 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:01:50.859185 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.859153 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq6k\" (UniqueName: \"kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.859367 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.859197 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.859457 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.859432 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.960554 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.960516 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.960809 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.960598 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq6k\" (UniqueName: \"kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.960809 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.960631 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.961311 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.961280 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:50.963213 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:50.963191 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:51.018784 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:51.018699 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq6k\" (UniqueName: \"kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k\") pod \"maas-api-revoke-sub-452b6d7b-8x56p\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:51.052891 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:51.052862 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:51.426098 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:51.426068 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p"] Nov 02 19:01:51.428188 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:01:51.428159 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ba040d_87f3_43f9_8c05_7d1dae346d67.slice/crio-14ef62b5a27fe76f5e81de9c647d6f703348cb55971e2a5f07c0953319b1ef28 WatchSource:0}: Error finding container 14ef62b5a27fe76f5e81de9c647d6f703348cb55971e2a5f07c0953319b1ef28: Status 404 returned error can't find the container with id 14ef62b5a27fe76f5e81de9c647d6f703348cb55971e2a5f07c0953319b1ef28 Nov 02 19:01:52.057601 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:52.057507 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" event={"ID":"e0ba040d-87f3-43f9-8c05-7d1dae346d67","Type":"ContainerStarted","Data":"7da2c21fd3df243ef37e0d720bffa1c632a1d70a176657d9a601f0aff02ff78e"} Nov 02 19:01:52.057601 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:52.057560 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" event={"ID":"e0ba040d-87f3-43f9-8c05-7d1dae346d67","Type":"ContainerStarted","Data":"14ef62b5a27fe76f5e81de9c647d6f703348cb55971e2a5f07c0953319b1ef28"} Nov 02 19:01:52.175003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:52.174946 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" podStartSLOduration=2.174931494 podStartE2EDuration="2.174931494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:01:52.173455126 +0000 UTC m=+5899.246982328" watchObservedRunningTime="2026-11-02 19:01:52.174931494 +0000 UTC m=+5899.248458761" Nov 02 19:01:52.238376 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:52.238340 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:01:53.067424 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:53.067381 5647 generic.go:356] "Generic (PLEG): container finished" podID="e0ba040d-87f3-43f9-8c05-7d1dae346d67" containerID="7da2c21fd3df243ef37e0d720bffa1c632a1d70a176657d9a601f0aff02ff78e" exitCode=0 Nov 02 19:01:53.068023 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:53.067481 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" event={"ID":"e0ba040d-87f3-43f9-8c05-7d1dae346d67","Type":"ContainerDied","Data":"7da2c21fd3df243ef37e0d720bffa1c632a1d70a176657d9a601f0aff02ff78e"} Nov 02 19:01:54.959638 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:54.959600 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:55.112928 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.112885 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca\") pod \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " Nov 02 19:01:55.113261 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.113235 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca\") pod \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " Nov 02 19:01:55.113482 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.113460 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k\" (UniqueName: \"kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k\") pod \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\" (UID: \"e0ba040d-87f3-43f9-8c05-7d1dae346d67\") " Nov 02 19:01:55.115277 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.115250 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca" pod "e0ba040d-87f3-43f9-8c05-7d1dae346d67" (UID: "e0ba040d-87f3-43f9-8c05-7d1dae346d67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:01:55.241783 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.241653 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca" pod "e0ba040d-87f3-43f9-8c05-7d1dae346d67" (UID: "e0ba040d-87f3-43f9-8c05-7d1dae346d67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:01:55.242363 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.242328 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k" pod "e0ba040d-87f3-43f9-8c05-7d1dae346d67" (UID: "e0ba040d-87f3-43f9-8c05-7d1dae346d67"). InnerVolumeSpecName "kube-api-access-bmq6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:01:55.242636 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.242465 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:01:55.247066 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.246926 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0ba040d-87f3-43f9-8c05-7d1dae346d67-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:01:55.291781 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.291714 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" event={"ID":"e0ba040d-87f3-43f9-8c05-7d1dae346d67","Type":"ContainerDied","Data":"14ef62b5a27fe76f5e81de9c647d6f703348cb55971e2a5f07c0953319b1ef28"} Nov 02 19:01:55.292095 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.292073 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14ef62b5a27fe76f5e81de9c647d6f703348cb55971e2a5f07c0953319b1ef28" Nov 02 19:01:55.292298 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.292283 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p" Nov 02 19:01:55.348475 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:01:55.348424 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmq6k\" (UniqueName: \"kubernetes.io/projected/e0ba040d-87f3-43f9-8c05-7d1dae346d67-kube-api-access-bmq6k\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:05.947039 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:02:05.946999 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:02:07.235394 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:07.235359 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:02:12.324604 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.324566 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2"] Nov 02 19:02:12.325229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.325049 5647 cpu_manager.go:414] "containerMap: removing container" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:02:12.325229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.325063 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:02:12.325229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.325100 5647 cpu_manager.go:414] "containerMap: removing container" podUID="e0ba040d-87f3-43f9-8c05-7d1dae346d67" containerName="revoke-keys" Nov 02 19:02:12.325229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.325106 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ba040d-87f3-43f9-8c05-7d1dae346d67" containerName="revoke-keys" Nov 02 19:02:12.325229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.325180 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5739c2-992f-4067-aac2-971c17c3d676" containerName="cleanup" Nov 02 19:02:12.325229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.325194 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ba040d-87f3-43f9-8c05-7d1dae346d67" containerName="revoke-keys" Nov 02 19:02:12.328603 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.328576 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.382336 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.382305 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:02:12.384482 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.384449 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2"] Nov 02 19:02:12.397859 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.397831 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfkq\" (UniqueName: \"kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.398065 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.398044 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.398146 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.398081 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.432949 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.432912 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:02:12.498953 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.498918 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkq\" (UniqueName: \"kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.499150 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.499059 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.499150 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.499081 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.499830 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.499811 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.501634 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.501609 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.557140 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.557113 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfkq\" (UniqueName: \"kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq\") pod \"maas-api-revoke-sub-3e31a506-8bxg2\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:12.641448 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:12.641409 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:13.009761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:13.009719 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2"] Nov 02 19:02:13.012076 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:02:13.012032 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod342aa88b_0ba9_4f9e_8089_774f198067a8.slice/crio-f46e662773760ce34c3ee6097beb49de5909fc4aa4a94bc2b6e7bfe50879d272 WatchSource:0}: Error finding container f46e662773760ce34c3ee6097beb49de5909fc4aa4a94bc2b6e7bfe50879d272: Status 404 returned error can't find the container with id f46e662773760ce34c3ee6097beb49de5909fc4aa4a94bc2b6e7bfe50879d272 Nov 02 19:02:13.417768 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:13.417710 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" event={"ID":"342aa88b-0ba9-4f9e-8089-774f198067a8","Type":"ContainerStarted","Data":"36af82c586b0869992deef437eb218d7d2789ff46872470f039ecce909c9a198"} Nov 02 19:02:13.418277 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:13.417780 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" event={"ID":"342aa88b-0ba9-4f9e-8089-774f198067a8","Type":"ContainerStarted","Data":"f46e662773760ce34c3ee6097beb49de5909fc4aa4a94bc2b6e7bfe50879d272"} Nov 02 19:02:13.534698 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:13.534634 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" podStartSLOduration=1.534619714 podStartE2EDuration="1.534619714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:02:13.533418164 +0000 UTC m=+5920.606945366" watchObservedRunningTime="2026-11-02 19:02:13.534619714 +0000 UTC m=+5920.608146985" Nov 02 19:02:14.429463 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:14.429415 5647 generic.go:356] "Generic (PLEG): container finished" podID="342aa88b-0ba9-4f9e-8089-774f198067a8" containerID="36af82c586b0869992deef437eb218d7d2789ff46872470f039ecce909c9a198" exitCode=0 Nov 02 19:02:14.430043 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:14.429537 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" event={"ID":"342aa88b-0ba9-4f9e-8089-774f198067a8","Type":"ContainerDied","Data":"36af82c586b0869992deef437eb218d7d2789ff46872470f039ecce909c9a198"} Nov 02 19:02:16.274213 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.243960 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:16.415066 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.415031 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca\") pod \"342aa88b-0ba9-4f9e-8089-774f198067a8\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " Nov 02 19:02:16.415441 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.415420 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca\") pod \"342aa88b-0ba9-4f9e-8089-774f198067a8\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " Nov 02 19:02:16.415607 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.415590 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq\" (UniqueName: \"kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq\") pod \"342aa88b-0ba9-4f9e-8089-774f198067a8\" (UID: \"342aa88b-0ba9-4f9e-8089-774f198067a8\") " Nov 02 19:02:16.531782 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.501867 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca" pod "342aa88b-0ba9-4f9e-8089-774f198067a8" (UID: "342aa88b-0ba9-4f9e-8089-774f198067a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:02:16.592788 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.592426 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" event={"ID":"342aa88b-0ba9-4f9e-8089-774f198067a8","Type":"ContainerDied","Data":"f46e662773760ce34c3ee6097beb49de5909fc4aa4a94bc2b6e7bfe50879d272"} Nov 02 19:02:16.592788 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.592495 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f46e662773760ce34c3ee6097beb49de5909fc4aa4a94bc2b6e7bfe50879d272" Nov 02 19:02:16.592788 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.592672 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2" Nov 02 19:02:16.601631 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.601598 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:16.663979 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.663923 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq" pod "342aa88b-0ba9-4f9e-8089-774f198067a8" (UID: "342aa88b-0ba9-4f9e-8089-774f198067a8"). InnerVolumeSpecName "kube-api-access-dqfkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:02:16.692042 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.691954 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca" pod "342aa88b-0ba9-4f9e-8089-774f198067a8" (UID: "342aa88b-0ba9-4f9e-8089-774f198067a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:02:16.718762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.718414 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/342aa88b-0ba9-4f9e-8089-774f198067a8-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:16.718762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:16.718454 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqfkq\" (UniqueName: \"kubernetes.io/projected/342aa88b-0ba9-4f9e-8089-774f198067a8-kube-api-access-dqfkq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:18.958051 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:18.958014 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:02:30.347439 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:30.347396 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:02:33.093117 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:33.093077 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:02:33.093798 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:33.093765 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:02:33.491821 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:33.488823 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:02:33.491821 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:33.488904 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:02:35.913832 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:35.913797 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926"] Nov 02 19:02:35.914295 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:35.914281 5647 cpu_manager.go:414] "containerMap: removing container" podUID="342aa88b-0ba9-4f9e-8089-774f198067a8" containerName="revoke-keys" Nov 02 19:02:35.914348 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:35.914297 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="342aa88b-0ba9-4f9e-8089-774f198067a8" containerName="revoke-keys" Nov 02 19:02:35.914387 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:35.914367 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="342aa88b-0ba9-4f9e-8089-774f198067a8" containerName="revoke-keys" Nov 02 19:02:35.917770 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:35.917749 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:35.949953 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:02:35.949917 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:02:35.974274 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:35.974233 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926"] Nov 02 19:02:36.009332 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.009291 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.009332 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.009333 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.009606 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.009474 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkxb\" (UniqueName: \"kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.022627 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.022595 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:02:36.022799 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.022775 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:02:36.111009 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.110970 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.111198 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.111015 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.111198 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.111094 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkxb\" (UniqueName: \"kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.111758 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.111714 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.113639 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.113617 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.169518 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.169442 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkxb\" (UniqueName: \"kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb\") pod \"maas-api-revoke-sub-01fda30b-wj926\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.235123 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.235087 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:36.605780 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.605749 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926"] Nov 02 19:02:36.607913 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:02:36.607883 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914e84b2_c55a_4a4c_8a96_65c37a9c55ad.slice/crio-dc26da2a094e41b6f2de154756bf22e1b9fe39725e0909c047b90e1496c7f6b9 WatchSource:0}: Error finding container dc26da2a094e41b6f2de154756bf22e1b9fe39725e0909c047b90e1496c7f6b9: Status 404 returned error can't find the container with id dc26da2a094e41b6f2de154756bf22e1b9fe39725e0909c047b90e1496c7f6b9 Nov 02 19:02:36.752712 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:36.752678 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" event={"ID":"914e84b2-c55a-4a4c-8a96-65c37a9c55ad","Type":"ContainerStarted","Data":"dc26da2a094e41b6f2de154756bf22e1b9fe39725e0909c047b90e1496c7f6b9"} Nov 02 19:02:37.759516 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:37.759425 5647 generic.go:356] "Generic (PLEG): container finished" podID="914e84b2-c55a-4a4c-8a96-65c37a9c55ad" containerID="83a6482ff3c6c3ff6e0e6304a3be8c8606a8d361aad14034378d571d6c3bb93b" exitCode=0 Nov 02 19:02:37.759932 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:37.759523 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" event={"ID":"914e84b2-c55a-4a4c-8a96-65c37a9c55ad","Type":"ContainerDied","Data":"83a6482ff3c6c3ff6e0e6304a3be8c8606a8d361aad14034378d571d6c3bb93b"} Nov 02 19:02:38.357510 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:38.357474 5647 scope.go:122] "RemoveContainer" containerID="b3def1b250102277816d02c565195332d0175cf9c7d70730767c502ce95d891e" Nov 02 19:02:38.366551 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:38.366530 5647 scope.go:122] "RemoveContainer" containerID="2bd6d6f2dfb01f8fda4af32d69b41dd5b5a82167513404506b4489d56996b615" Nov 02 19:02:38.910085 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:38.910057 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:39.042688 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.042605 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca\") pod \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " Nov 02 19:02:39.042688 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.042642 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb\" (UniqueName: \"kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb\") pod \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " Nov 02 19:02:39.042947 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.042791 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca\") pod \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\" (UID: \"914e84b2-c55a-4a4c-8a96-65c37a9c55ad\") " Nov 02 19:02:39.043048 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.043026 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca" pod "914e84b2-c55a-4a4c-8a96-65c37a9c55ad" (UID: "914e84b2-c55a-4a4c-8a96-65c37a9c55ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:02:39.043261 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.043236 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:39.045001 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.044976 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb" pod "914e84b2-c55a-4a4c-8a96-65c37a9c55ad" (UID: "914e84b2-c55a-4a4c-8a96-65c37a9c55ad"). InnerVolumeSpecName "kube-api-access-prkxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:02:39.045116 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.045033 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca" pod "914e84b2-c55a-4a4c-8a96-65c37a9c55ad" (UID: "914e84b2-c55a-4a4c-8a96-65c37a9c55ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:02:39.144535 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.144504 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkxb\" (UniqueName: \"kubernetes.io/projected/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-kube-api-access-prkxb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:39.144535 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.144537 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/914e84b2-c55a-4a4c-8a96-65c37a9c55ad-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:02:39.770118 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.770089 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" Nov 02 19:02:39.770118 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.770092 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926" event={"ID":"914e84b2-c55a-4a4c-8a96-65c37a9c55ad","Type":"ContainerDied","Data":"dc26da2a094e41b6f2de154756bf22e1b9fe39725e0909c047b90e1496c7f6b9"} Nov 02 19:02:39.770346 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:39.770138 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc26da2a094e41b6f2de154756bf22e1b9fe39725e0909c047b90e1496c7f6b9" Nov 02 19:02:48.662712 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:02:48.662623 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:03:05.947662 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:03:05.947622 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:03:21.920405 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:21.920369 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw"] Nov 02 19:03:21.921049 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:21.920867 5647 cpu_manager.go:414] "containerMap: removing container" podUID="914e84b2-c55a-4a4c-8a96-65c37a9c55ad" containerName="revoke-keys" Nov 02 19:03:21.921049 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:21.920905 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914e84b2-c55a-4a4c-8a96-65c37a9c55ad" containerName="revoke-keys" Nov 02 19:03:21.921049 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:21.921010 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="914e84b2-c55a-4a4c-8a96-65c37a9c55ad" containerName="revoke-keys" Nov 02 19:03:21.924164 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:21.924148 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:21.978677 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:21.978642 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 19:03:22.029185 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.029154 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kx74g\"" Nov 02 19:03:22.030865 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.030842 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw"] Nov 02 19:03:22.094216 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.094177 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743fb649-948e-4a93-80d3-be11fd66e728-plugins-config-volume\") pod \"payload-pre-processing-5fc8898f46-mmzfw\" (UID: \"743fb649-948e-4a93-80d3-be11fd66e728\") " pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.094417 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.094246 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh4lb\" (UniqueName: \"kubernetes.io/projected/743fb649-948e-4a93-80d3-be11fd66e728-kube-api-access-dh4lb\") pod \"payload-pre-processing-5fc8898f46-mmzfw\" (UID: \"743fb649-948e-4a93-80d3-be11fd66e728\") " pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.195566 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.195489 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743fb649-948e-4a93-80d3-be11fd66e728-plugins-config-volume\") pod \"payload-pre-processing-5fc8898f46-mmzfw\" (UID: \"743fb649-948e-4a93-80d3-be11fd66e728\") " pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.195566 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.195545 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh4lb\" (UniqueName: \"kubernetes.io/projected/743fb649-948e-4a93-80d3-be11fd66e728-kube-api-access-dh4lb\") pod \"payload-pre-processing-5fc8898f46-mmzfw\" (UID: \"743fb649-948e-4a93-80d3-be11fd66e728\") " pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.196296 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.196275 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743fb649-948e-4a93-80d3-be11fd66e728-plugins-config-volume\") pod \"payload-pre-processing-5fc8898f46-mmzfw\" (UID: \"743fb649-948e-4a93-80d3-be11fd66e728\") " pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.253618 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.253583 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh4lb\" (UniqueName: \"kubernetes.io/projected/743fb649-948e-4a93-80d3-be11fd66e728-kube-api-access-dh4lb\") pod \"payload-pre-processing-5fc8898f46-mmzfw\" (UID: \"743fb649-948e-4a93-80d3-be11fd66e728\") " pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.536929 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.536837 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:22.907132 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.907083 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw"] Nov 02 19:03:22.909867 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:03:22.909838 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743fb649_948e_4a93_80d3_be11fd66e728.slice/crio-f378a8978c4ac2071e4ee5c28345ca4c736e1aeb81c53c5c4ee5717859ceac2f WatchSource:0}: Error finding container f378a8978c4ac2071e4ee5c28345ca4c736e1aeb81c53c5c4ee5717859ceac2f: Status 404 returned error can't find the container with id f378a8978c4ac2071e4ee5c28345ca4c736e1aeb81c53c5c4ee5717859ceac2f Nov 02 19:03:22.990469 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.990431 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" event={"ID":"743fb649-948e-4a93-80d3-be11fd66e728","Type":"ContainerStarted","Data":"79e59d3e2be7daa3d2ab8bdb3e6577d91da98fd9293c7a267277d5ea6c91e6b8"} Nov 02 19:03:22.991003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.990492 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" event={"ID":"743fb649-948e-4a93-80d3-be11fd66e728","Type":"ContainerStarted","Data":"f378a8978c4ac2071e4ee5c28345ca4c736e1aeb81c53c5c4ee5717859ceac2f"} Nov 02 19:03:22.991003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:22.990646 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:23.107228 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:23.107169 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" podStartSLOduration=2.107154588 podStartE2EDuration="2.107154588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:03:23.105755898 +0000 UTC m=+5990.179283093" watchObservedRunningTime="2026-11-02 19:03:23.107154588 +0000 UTC m=+5990.180681791" Nov 02 19:03:34.000125 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:34.000088 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5fc8898f46-mmzfw" Nov 02 19:03:35.046749 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:35.046694 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:03:35.053418 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:35.053396 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:03:35.946338 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:03:35.946300 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:03:38.390637 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:38.390606 5647 scope.go:122] "RemoveContainer" containerID="370a754c355ac1f7c35f7f70c215f847f516119509892657a2da60a6cc34eae0" Nov 02 19:03:42.179860 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:03:42.179830 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:04:05.946710 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:04:05.946670 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:04:35.947147 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:04:35.947063 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:04:44.046636 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:04:44.046600 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn"] Nov 02 19:04:44.096934 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:04:44.096899 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7f3bd92-g4rvn"] Nov 02 19:04:45.079717 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:04:45.079683 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3546b94b-9c02-45d0-becc-64034f027be3" path="/var/lib/kubelet/pods/3546b94b-9c02-45d0-becc-64034f027be3/volumes" Nov 02 19:05:05.947999 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:05:05.947946 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:05:14.450210 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.450176 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9"] Nov 02 19:05:14.453897 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.453881 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.508977 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.508941 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:05:14.559461 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.559430 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:05:14.562394 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.562365 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9"] Nov 02 19:05:14.587343 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.587310 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.587532 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.587358 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.587532 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.587449 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2nfb\" (UniqueName: \"kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.688665 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.688629 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.688665 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.688680 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.688958 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.688726 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2nfb\" (UniqueName: \"kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.689458 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.689437 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.691231 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.691214 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.746805 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.746696 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2nfb\" (UniqueName: \"kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb\") pod \"maas-api-revoke-sub-e2547622-wg5m9\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:14.767006 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:14.766976 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:15.137329 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:15.137305 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9"] Nov 02 19:05:15.139512 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:05:15.139482 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9ab9c2_f452_4e93_b43f_26800dcd2625.slice/crio-73eae725f80ab73f42b00420012651fa409cfb5fcd52c3ce2b3ac2dd3a2de6cd WatchSource:0}: Error finding container 73eae725f80ab73f42b00420012651fa409cfb5fcd52c3ce2b3ac2dd3a2de6cd: Status 404 returned error can't find the container with id 73eae725f80ab73f42b00420012651fa409cfb5fcd52c3ce2b3ac2dd3a2de6cd Nov 02 19:05:15.604185 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:15.604144 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" event={"ID":"0f9ab9c2-f452-4e93-b43f-26800dcd2625","Type":"ContainerStarted","Data":"08cf5005aeb06e482d8037b091a34c0ff3cb7ca41f6832fb809e52161cd982fe"} Nov 02 19:05:15.604750 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:15.604205 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" event={"ID":"0f9ab9c2-f452-4e93-b43f-26800dcd2625","Type":"ContainerStarted","Data":"73eae725f80ab73f42b00420012651fa409cfb5fcd52c3ce2b3ac2dd3a2de6cd"} Nov 02 19:05:15.720151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:15.720049 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" podStartSLOduration=1.720034932 podStartE2EDuration="1.720034932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:05:15.719101685 +0000 UTC m=+6102.792628892" watchObservedRunningTime="2026-11-02 19:05:15.720034932 +0000 UTC m=+6102.793562132" Nov 02 19:05:16.611226 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:16.611193 5647 generic.go:356] "Generic (PLEG): container finished" podID="0f9ab9c2-f452-4e93-b43f-26800dcd2625" containerID="08cf5005aeb06e482d8037b091a34c0ff3cb7ca41f6832fb809e52161cd982fe" exitCode=0 Nov 02 19:05:16.611651 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:16.611271 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" event={"ID":"0f9ab9c2-f452-4e93-b43f-26800dcd2625","Type":"ContainerDied","Data":"08cf5005aeb06e482d8037b091a34c0ff3cb7ca41f6832fb809e52161cd982fe"} Nov 02 19:05:17.748571 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.748547 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:17.922041 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.921958 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb\" (UniqueName: \"kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb\") pod \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " Nov 02 19:05:17.922041 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.922011 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca\") pod \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " Nov 02 19:05:17.922309 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.922052 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca\") pod \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\" (UID: \"0f9ab9c2-f452-4e93-b43f-26800dcd2625\") " Nov 02 19:05:17.922382 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.922354 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca" pod "0f9ab9c2-f452-4e93-b43f-26800dcd2625" (UID: "0f9ab9c2-f452-4e93-b43f-26800dcd2625"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:05:17.922518 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.922502 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:05:17.924287 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.924259 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca" pod "0f9ab9c2-f452-4e93-b43f-26800dcd2625" (UID: "0f9ab9c2-f452-4e93-b43f-26800dcd2625"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:05:17.924287 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:17.924275 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb" pod "0f9ab9c2-f452-4e93-b43f-26800dcd2625" (UID: "0f9ab9c2-f452-4e93-b43f-26800dcd2625"). InnerVolumeSpecName "kube-api-access-k2nfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:05:18.023714 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:18.023663 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2nfb\" (UniqueName: \"kubernetes.io/projected/0f9ab9c2-f452-4e93-b43f-26800dcd2625-kube-api-access-k2nfb\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:05:18.023714 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:18.023710 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9ab9c2-f452-4e93-b43f-26800dcd2625-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:05:18.623640 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:18.623612 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" Nov 02 19:05:18.623874 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:18.623611 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9" event={"ID":"0f9ab9c2-f452-4e93-b43f-26800dcd2625","Type":"ContainerDied","Data":"73eae725f80ab73f42b00420012651fa409cfb5fcd52c3ce2b3ac2dd3a2de6cd"} Nov 02 19:05:18.623874 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:18.623770 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73eae725f80ab73f42b00420012651fa409cfb5fcd52c3ce2b3ac2dd3a2de6cd" Nov 02 19:05:28.957228 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:28.957190 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:05:35.947413 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:05:35.947369 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:05:39.482369 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:39.482333 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:05:45.038957 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.038857 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b"] Nov 02 19:05:45.039538 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.039516 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0f9ab9c2-f452-4e93-b43f-26800dcd2625" containerName="revoke-keys" Nov 02 19:05:45.039629 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.039542 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9ab9c2-f452-4e93-b43f-26800dcd2625" containerName="revoke-keys" Nov 02 19:05:45.039774 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.039665 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9ab9c2-f452-4e93-b43f-26800dcd2625" containerName="revoke-keys" Nov 02 19:05:45.043201 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.043176 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.099413 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.099382 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b"] Nov 02 19:05:45.146911 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.146870 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:05:45.147117 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.146961 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:05:45.209870 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.209831 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.210088 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.209976 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.210171 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.210086 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrml\" (UniqueName: \"kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.311006 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.310913 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.311177 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.311010 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.311177 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.311051 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrml\" (UniqueName: \"kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.311631 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.311608 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.313636 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.313608 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.369408 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.369374 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrml\" (UniqueName: \"kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml\") pod \"maas-api-revoke-sub-c1e271a9-mzg6b\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:45.657232 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:45.657194 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:46.030179 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:46.030151 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b"] Nov 02 19:05:46.033044 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:05:46.033009 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde19e992_faa2_4a8f_8f2c_815c2258766c.slice/crio-0f013dc1fbaba4c5e969d9e88bdc0b789894895ee85381177d1731b44430383d WatchSource:0}: Error finding container 0f013dc1fbaba4c5e969d9e88bdc0b789894895ee85381177d1731b44430383d: Status 404 returned error can't find the container with id 0f013dc1fbaba4c5e969d9e88bdc0b789894895ee85381177d1731b44430383d Nov 02 19:05:46.968686 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:46.968648 5647 generic.go:356] "Generic (PLEG): container finished" podID="de19e992-faa2-4a8f-8f2c-815c2258766c" containerID="4c1e8e9eaee1d4d87ecc78ecba8f43c2cb663f6a85391c0904c9ce7418afa15e" exitCode=0 Nov 02 19:05:46.969129 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:46.968696 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" event={"ID":"de19e992-faa2-4a8f-8f2c-815c2258766c","Type":"ContainerDied","Data":"4c1e8e9eaee1d4d87ecc78ecba8f43c2cb663f6a85391c0904c9ce7418afa15e"} Nov 02 19:05:46.969129 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:46.968767 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" event={"ID":"de19e992-faa2-4a8f-8f2c-815c2258766c","Type":"ContainerStarted","Data":"0f013dc1fbaba4c5e969d9e88bdc0b789894895ee85381177d1731b44430383d"} Nov 02 19:05:48.107522 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.107499 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:48.243115 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.243035 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml\" (UniqueName: \"kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml\") pod \"de19e992-faa2-4a8f-8f2c-815c2258766c\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " Nov 02 19:05:48.243283 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.243131 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca\") pod \"de19e992-faa2-4a8f-8f2c-815c2258766c\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " Nov 02 19:05:48.243283 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.243235 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca\") pod \"de19e992-faa2-4a8f-8f2c-815c2258766c\" (UID: \"de19e992-faa2-4a8f-8f2c-815c2258766c\") " Nov 02 19:05:48.244110 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.243672 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca" pod "de19e992-faa2-4a8f-8f2c-815c2258766c" (UID: "de19e992-faa2-4a8f-8f2c-815c2258766c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:05:48.244568 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.244537 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:05:48.245970 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.245949 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca" pod "de19e992-faa2-4a8f-8f2c-815c2258766c" (UID: "de19e992-faa2-4a8f-8f2c-815c2258766c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:05:48.246136 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.246121 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml" pod "de19e992-faa2-4a8f-8f2c-815c2258766c" (UID: "de19e992-faa2-4a8f-8f2c-815c2258766c"). InnerVolumeSpecName "kube-api-access-mhrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:05:48.345775 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.345722 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhrml\" (UniqueName: \"kubernetes.io/projected/de19e992-faa2-4a8f-8f2c-815c2258766c-kube-api-access-mhrml\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:05:48.345775 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.345770 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de19e992-faa2-4a8f-8f2c-815c2258766c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:05:48.979485 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.979450 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" Nov 02 19:05:48.979711 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.979487 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b" event={"ID":"de19e992-faa2-4a8f-8f2c-815c2258766c","Type":"ContainerDied","Data":"0f013dc1fbaba4c5e969d9e88bdc0b789894895ee85381177d1731b44430383d"} Nov 02 19:05:48.979711 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:48.979532 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f013dc1fbaba4c5e969d9e88bdc0b789894895ee85381177d1731b44430383d" Nov 02 19:05:57.781600 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:05:57.781562 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:06:05.946894 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:06:05.946853 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:06:08.334630 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:08.334589 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:06:09.763279 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.763249 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw"] Nov 02 19:06:09.763720 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.763711 5647 cpu_manager.go:414] "containerMap: removing container" podUID="de19e992-faa2-4a8f-8f2c-815c2258766c" containerName="revoke-keys" Nov 02 19:06:09.763803 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.763724 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de19e992-faa2-4a8f-8f2c-815c2258766c" containerName="revoke-keys" Nov 02 19:06:09.763865 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.763853 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="de19e992-faa2-4a8f-8f2c-815c2258766c" containerName="revoke-keys" Nov 02 19:06:09.767333 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.767311 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:09.822128 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.822081 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:06:09.872407 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.872371 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:06:09.874981 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.874950 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw"] Nov 02 19:06:09.904215 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.904179 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:09.904389 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.904242 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:09.904389 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:09.904345 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9twln\" (UniqueName: \"kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.005542 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.005506 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.005806 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.005561 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.005806 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.005592 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9twln\" (UniqueName: \"kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.006177 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.006154 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.008192 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.008167 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.063458 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.063386 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9twln\" (UniqueName: \"kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln\") pod \"maas-api-revoke-sub-2e1f5b5f-bqsmw\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.081325 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.081295 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:10.452235 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:10.452210 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw"] Nov 02 19:06:10.454233 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:06:10.454206 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1e65fa_31ba_40ee_b273_730827a223ff.slice/crio-7b932ff13cf2a21c5f587e6c4e2c4767c2219fdae63fa81e034b2f4333fc2b9b WatchSource:0}: Error finding container 7b932ff13cf2a21c5f587e6c4e2c4767c2219fdae63fa81e034b2f4333fc2b9b: Status 404 returned error can't find the container with id 7b932ff13cf2a21c5f587e6c4e2c4767c2219fdae63fa81e034b2f4333fc2b9b Nov 02 19:06:11.127021 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:11.126990 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" event={"ID":"ad1e65fa-31ba-40ee-b273-730827a223ff","Type":"ContainerStarted","Data":"c325517b5e18fc6d7dcfc129ac2288c5ec360292f84f24e149c56c646f7de135"} Nov 02 19:06:11.127466 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:11.127041 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" event={"ID":"ad1e65fa-31ba-40ee-b273-730827a223ff","Type":"ContainerStarted","Data":"7b932ff13cf2a21c5f587e6c4e2c4767c2219fdae63fa81e034b2f4333fc2b9b"} Nov 02 19:06:11.244097 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:11.244043 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" podStartSLOduration=2.244025912 podStartE2EDuration="2.244025912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:06:11.24253127 +0000 UTC m=+6158.316058474" watchObservedRunningTime="2026-11-02 19:06:11.244025912 +0000 UTC m=+6158.317553115" Nov 02 19:06:12.133728 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:12.133693 5647 generic.go:356] "Generic (PLEG): container finished" podID="ad1e65fa-31ba-40ee-b273-730827a223ff" containerID="c325517b5e18fc6d7dcfc129ac2288c5ec360292f84f24e149c56c646f7de135" exitCode=0 Nov 02 19:06:12.134155 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:12.133777 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" event={"ID":"ad1e65fa-31ba-40ee-b273-730827a223ff","Type":"ContainerDied","Data":"c325517b5e18fc6d7dcfc129ac2288c5ec360292f84f24e149c56c646f7de135"} Nov 02 19:06:13.277622 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.277590 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:13.441608 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.441524 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca\") pod \"ad1e65fa-31ba-40ee-b273-730827a223ff\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " Nov 02 19:06:13.441608 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.441586 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln\" (UniqueName: \"kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln\") pod \"ad1e65fa-31ba-40ee-b273-730827a223ff\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " Nov 02 19:06:13.441880 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.441663 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca\") pod \"ad1e65fa-31ba-40ee-b273-730827a223ff\" (UID: \"ad1e65fa-31ba-40ee-b273-730827a223ff\") " Nov 02 19:06:13.441977 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.441952 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca" pod "ad1e65fa-31ba-40ee-b273-730827a223ff" (UID: "ad1e65fa-31ba-40ee-b273-730827a223ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:06:13.442153 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.442127 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:06:13.444003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.443977 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca" pod "ad1e65fa-31ba-40ee-b273-730827a223ff" (UID: "ad1e65fa-31ba-40ee-b273-730827a223ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:06:13.444003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.443994 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln" pod "ad1e65fa-31ba-40ee-b273-730827a223ff" (UID: "ad1e65fa-31ba-40ee-b273-730827a223ff"). InnerVolumeSpecName "kube-api-access-9twln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:06:13.543700 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.543667 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9twln\" (UniqueName: \"kubernetes.io/projected/ad1e65fa-31ba-40ee-b273-730827a223ff-kube-api-access-9twln\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:06:13.543700 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:13.543696 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad1e65fa-31ba-40ee-b273-730827a223ff-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:06:14.145174 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:14.145145 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" Nov 02 19:06:14.145391 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:14.145175 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw" event={"ID":"ad1e65fa-31ba-40ee-b273-730827a223ff","Type":"ContainerDied","Data":"7b932ff13cf2a21c5f587e6c4e2c4767c2219fdae63fa81e034b2f4333fc2b9b"} Nov 02 19:06:14.145391 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:14.145224 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b932ff13cf2a21c5f587e6c4e2c4767c2219fdae63fa81e034b2f4333fc2b9b" Nov 02 19:06:22.192189 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.192158 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc"] Nov 02 19:06:22.192792 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.192722 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ad1e65fa-31ba-40ee-b273-730827a223ff" containerName="revoke-keys" Nov 02 19:06:22.192792 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.192791 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1e65fa-31ba-40ee-b273-730827a223ff" containerName="revoke-keys" Nov 02 19:06:22.193035 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.192948 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1e65fa-31ba-40ee-b273-730827a223ff" containerName="revoke-keys" Nov 02 19:06:22.196392 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.196367 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.249824 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.249762 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:06:22.300394 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.300355 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:06:22.302212 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.302189 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc"] Nov 02 19:06:22.336097 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.336068 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.336246 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.336114 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.336329 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.336306 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdw7\" (UniqueName: \"kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.437647 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.437607 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdw7\" (UniqueName: \"kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.437904 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.437674 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.437904 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.437849 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.438418 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.438391 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.440364 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.440347 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.495470 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.495395 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdw7\" (UniqueName: \"kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7\") pod \"maas-api-revoke-sub-ec843018-4x4tc\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.511399 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.511354 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:22.885955 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:22.885928 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc"] Nov 02 19:06:22.888019 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:06:22.887980 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c24d7e_5c00_4f05_a3d8_ef2bfdd1d5d7.slice/crio-cb95e8ae7c2f9a78211c84503db4cab84154a23d61ed1de3f9fc0f5dcf3c17ea WatchSource:0}: Error finding container cb95e8ae7c2f9a78211c84503db4cab84154a23d61ed1de3f9fc0f5dcf3c17ea: Status 404 returned error can't find the container with id cb95e8ae7c2f9a78211c84503db4cab84154a23d61ed1de3f9fc0f5dcf3c17ea Nov 02 19:06:23.193114 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:23.193025 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" event={"ID":"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7","Type":"ContainerStarted","Data":"cb95e8ae7c2f9a78211c84503db4cab84154a23d61ed1de3f9fc0f5dcf3c17ea"} Nov 02 19:06:24.199429 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:24.199397 5647 generic.go:356] "Generic (PLEG): container finished" podID="c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" containerID="7deb412af042ed7a0d0e987dcc7d6e02e245d122b668d67470ae443334ec13c1" exitCode=0 Nov 02 19:06:24.199867 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:24.199486 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" event={"ID":"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7","Type":"ContainerDied","Data":"7deb412af042ed7a0d0e987dcc7d6e02e245d122b668d67470ae443334ec13c1"} Nov 02 19:06:25.346028 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.346002 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:25.369181 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.369154 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7\" (UniqueName: \"kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7\") pod \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " Nov 02 19:06:25.369365 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.369192 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca\") pod \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " Nov 02 19:06:25.369365 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.369228 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca\") pod \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\" (UID: \"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7\") " Nov 02 19:06:25.369591 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.369568 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca" pod "c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" (UID: "c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:06:25.371443 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.371415 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca" pod "c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" (UID: "c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:06:25.371580 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.371559 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7" pod "c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" (UID: "c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7"). InnerVolumeSpecName "kube-api-access-nhdw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:06:25.470518 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.470435 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdw7\" (UniqueName: \"kubernetes.io/projected/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-kube-api-access-nhdw7\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:06:25.470518 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.470474 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:06:25.470518 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:25.470493 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:06:26.078751 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:26.078703 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:06:26.211050 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:26.211016 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" event={"ID":"c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7","Type":"ContainerDied","Data":"cb95e8ae7c2f9a78211c84503db4cab84154a23d61ed1de3f9fc0f5dcf3c17ea"} Nov 02 19:06:26.211229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:26.211054 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc" Nov 02 19:06:26.211229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:26.211068 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb95e8ae7c2f9a78211c84503db4cab84154a23d61ed1de3f9fc0f5dcf3c17ea" Nov 02 19:06:35.946581 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:06:35.946541 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:06:38.425951 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:38.425917 5647 scope.go:122] "RemoveContainer" containerID="774b0060f1533739461d669d6b216a186fe63d943d1978445ab8cfc1e8916619" Nov 02 19:06:39.366787 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:39.366753 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:06:55.046391 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:55.046354 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p"] Nov 02 19:06:55.097070 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:55.097033 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-8x56p"] Nov 02 19:06:57.079852 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.079816 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ba040d-87f3-43f9-8c05-7d1dae346d67" path="/var/lib/kubelet/pods/e0ba040d-87f3-43f9-8c05-7d1dae346d67/volumes" Nov 02 19:06:57.460524 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.460487 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt"] Nov 02 19:06:57.460947 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.460933 5647 cpu_manager.go:414] "containerMap: removing container" podUID="c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" containerName="revoke-keys" Nov 02 19:06:57.460999 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.460948 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" containerName="revoke-keys" Nov 02 19:06:57.461042 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.461023 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" containerName="revoke-keys" Nov 02 19:06:57.464107 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.464091 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.465795 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.465775 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.465898 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.465806 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ccq\" (UniqueName: \"kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.465999 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.465979 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.566560 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.566528 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.566560 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.566566 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ccq\" (UniqueName: \"kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.566791 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.566668 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.567241 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.567222 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.569334 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.569314 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:06:57.569517 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.569493 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:06:57.571285 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.571266 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt"] Nov 02 19:06:57.580534 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.580512 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.624699 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.624660 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ccq\" (UniqueName: \"kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq\") pod \"maas-api-revoke-sub-046e1343-b99dt\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:57.777305 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:57.777211 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:06:58.150057 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:58.150024 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt"] Nov 02 19:06:58.151617 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:06:58.151591 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283b2c0d_031c_45e6_bcf2_e5c99f1ae36e.slice/crio-d72479f68c978a798fc1580b6f2b43f958f321898df254e03ff07d533435d528 WatchSource:0}: Error finding container d72479f68c978a798fc1580b6f2b43f958f321898df254e03ff07d533435d528: Status 404 returned error can't find the container with id d72479f68c978a798fc1580b6f2b43f958f321898df254e03ff07d533435d528 Nov 02 19:06:58.437899 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:58.437853 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" event={"ID":"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e","Type":"ContainerStarted","Data":"30f6988771e7bdb47584193c443f35f72789b19b6b1c04f6d374e637ecadd2dc"} Nov 02 19:06:58.437899 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:58.437908 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" event={"ID":"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e","Type":"ContainerStarted","Data":"d72479f68c978a798fc1580b6f2b43f958f321898df254e03ff07d533435d528"} Nov 02 19:06:58.553059 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:58.553002 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" podStartSLOduration=1.552986398 podStartE2EDuration="1.552986398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:06:58.551682435 +0000 UTC m=+6205.625209630" watchObservedRunningTime="2026-11-02 19:06:58.552986398 +0000 UTC m=+6205.626513603" Nov 02 19:06:59.444861 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:59.444828 5647 generic.go:356] "Generic (PLEG): container finished" podID="283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" containerID="30f6988771e7bdb47584193c443f35f72789b19b6b1c04f6d374e637ecadd2dc" exitCode=0 Nov 02 19:06:59.445281 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:06:59.444917 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" event={"ID":"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e","Type":"ContainerDied","Data":"30f6988771e7bdb47584193c443f35f72789b19b6b1c04f6d374e637ecadd2dc"} Nov 02 19:07:00.583819 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.583794 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:07:00.589031 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.589007 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq\" (UniqueName: \"kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq\") pod \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " Nov 02 19:07:00.589165 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.589106 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca\") pod \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " Nov 02 19:07:00.589165 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.589132 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca\") pod \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\" (UID: \"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e\") " Nov 02 19:07:00.589473 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.589455 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca" pod "283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" (UID: "283b2c0d-031c-45e6-bcf2-e5c99f1ae36e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:07:00.591168 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.591138 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca" pod "283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" (UID: "283b2c0d-031c-45e6-bcf2-e5c99f1ae36e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:07:00.591299 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.591275 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq" pod "283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" (UID: "283b2c0d-031c-45e6-bcf2-e5c99f1ae36e"). InnerVolumeSpecName "kube-api-access-s4ccq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:07:00.690292 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.690258 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ccq\" (UniqueName: \"kubernetes.io/projected/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-kube-api-access-s4ccq\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:00.690292 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.690287 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:00.690292 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:00.690297 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:01.456544 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:01.456460 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" event={"ID":"283b2c0d-031c-45e6-bcf2-e5c99f1ae36e","Type":"ContainerDied","Data":"d72479f68c978a798fc1580b6f2b43f958f321898df254e03ff07d533435d528"} Nov 02 19:07:01.456544 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:01.456511 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d72479f68c978a798fc1580b6f2b43f958f321898df254e03ff07d533435d528" Nov 02 19:07:01.456544 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:01.456471 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt" Nov 02 19:07:05.947549 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:07:05.947509 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:07:08.204569 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.204532 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp"] Nov 02 19:07:08.205244 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.205224 5647 cpu_manager.go:414] "containerMap: removing container" podUID="283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" containerName="revoke-keys" Nov 02 19:07:08.205296 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.205249 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" containerName="revoke-keys" Nov 02 19:07:08.205426 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.205414 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" containerName="revoke-keys" Nov 02 19:07:08.209138 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.209121 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.261002 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.260966 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jr9\" (UniqueName: \"kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.261180 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.261046 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.261180 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.261147 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.263579 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.263549 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp"] Nov 02 19:07:08.311501 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.311462 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:07:08.311676 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.311519 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:07:08.362254 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.362226 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.362455 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.362321 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jr9\" (UniqueName: \"kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.362455 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.362375 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.363000 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.362973 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.364786 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.364764 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.419854 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.419819 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jr9\" (UniqueName: \"kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9\") pod \"maas-api-revoke-sub-d3fba056-trlwp\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.532146 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.532069 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:08.903892 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:08.903691 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp"] Nov 02 19:07:08.906345 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:07:08.906307 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7791d99_2f83_4249_8952_3e660405c723.slice/crio-9bc6e236e8a715505f7c3994eea59bc2a2801c305cdffdfc83b848b929df1379 WatchSource:0}: Error finding container 9bc6e236e8a715505f7c3994eea59bc2a2801c305cdffdfc83b848b929df1379: Status 404 returned error can't find the container with id 9bc6e236e8a715505f7c3994eea59bc2a2801c305cdffdfc83b848b929df1379 Nov 02 19:07:09.499693 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:09.499603 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" event={"ID":"d7791d99-2f83-4249-8952-3e660405c723","Type":"ContainerStarted","Data":"79f7ad411d36de3a86260d0a228079f3a7a7e43c8abd1a9394a0873a71d145f2"} Nov 02 19:07:09.499693 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:09.499690 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" event={"ID":"d7791d99-2f83-4249-8952-3e660405c723","Type":"ContainerStarted","Data":"9bc6e236e8a715505f7c3994eea59bc2a2801c305cdffdfc83b848b929df1379"} Nov 02 19:07:09.615043 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:09.614984 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" podStartSLOduration=1.6149719340000002 podStartE2EDuration="1.614971934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:07:09.61426043 +0000 UTC m=+6216.687787646" watchObservedRunningTime="2026-11-02 19:07:09.614971934 +0000 UTC m=+6216.688499137" Nov 02 19:07:10.271003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:10.270968 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:07:10.506783 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:10.506752 5647 generic.go:356] "Generic (PLEG): container finished" podID="d7791d99-2f83-4249-8952-3e660405c723" containerID="79f7ad411d36de3a86260d0a228079f3a7a7e43c8abd1a9394a0873a71d145f2" exitCode=0 Nov 02 19:07:10.507395 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:10.506788 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" event={"ID":"d7791d99-2f83-4249-8952-3e660405c723","Type":"ContainerDied","Data":"79f7ad411d36de3a86260d0a228079f3a7a7e43c8abd1a9394a0873a71d145f2"} Nov 02 19:07:11.984760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:11.959988 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:12.000140 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.000093 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca\") pod \"d7791d99-2f83-4249-8952-3e660405c723\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " Nov 02 19:07:12.000334 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.000165 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9\" (UniqueName: \"kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9\") pod \"d7791d99-2f83-4249-8952-3e660405c723\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " Nov 02 19:07:12.000334 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.000257 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca\") pod \"d7791d99-2f83-4249-8952-3e660405c723\" (UID: \"d7791d99-2f83-4249-8952-3e660405c723\") " Nov 02 19:07:12.000966 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.000909 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca" pod "d7791d99-2f83-4249-8952-3e660405c723" (UID: "d7791d99-2f83-4249-8952-3e660405c723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:07:12.027759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.026909 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9" pod "d7791d99-2f83-4249-8952-3e660405c723" (UID: "d7791d99-2f83-4249-8952-3e660405c723"). InnerVolumeSpecName "kube-api-access-q6jr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:07:12.032756 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.029297 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca" pod "d7791d99-2f83-4249-8952-3e660405c723" (UID: "d7791d99-2f83-4249-8952-3e660405c723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:07:12.102465 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.102363 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7791d99-2f83-4249-8952-3e660405c723-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:12.102465 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.102400 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jr9\" (UniqueName: \"kubernetes.io/projected/d7791d99-2f83-4249-8952-3e660405c723-kube-api-access-q6jr9\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:12.102465 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.102419 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7791d99-2f83-4249-8952-3e660405c723-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:12.550760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.548914 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" event={"ID":"d7791d99-2f83-4249-8952-3e660405c723","Type":"ContainerDied","Data":"9bc6e236e8a715505f7c3994eea59bc2a2801c305cdffdfc83b848b929df1379"} Nov 02 19:07:12.550760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.548982 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc6e236e8a715505f7c3994eea59bc2a2801c305cdffdfc83b848b929df1379" Nov 02 19:07:12.550760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:12.549066 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp" Nov 02 19:07:16.043673 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:16.043633 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2"] Nov 02 19:07:16.094203 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:16.094168 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8bxg2"] Nov 02 19:07:17.079937 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:17.079901 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="342aa88b-0ba9-4f9e-8089-774f198067a8" path="/var/lib/kubelet/pods/342aa88b-0ba9-4f9e-8089-774f198067a8/volumes" Nov 02 19:07:21.063835 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:21.063798 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:07:33.382890 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:33.382848 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:07:35.947641 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:07:35.947596 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:07:40.051712 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:40.051671 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926"] Nov 02 19:07:40.102270 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:40.102236 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fda30b-wj926"] Nov 02 19:07:41.080381 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:41.080340 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914e84b2-c55a-4a4c-8a96-65c37a9c55ad" path="/var/lib/kubelet/pods/914e84b2-c55a-4a4c-8a96-65c37a9c55ad/volumes" Nov 02 19:07:43.047051 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.047016 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4"] Nov 02 19:07:43.047813 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.047789 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d7791d99-2f83-4249-8952-3e660405c723" containerName="revoke-keys" Nov 02 19:07:43.047927 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.047817 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7791d99-2f83-4249-8952-3e660405c723" containerName="revoke-keys" Nov 02 19:07:43.047990 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.047941 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7791d99-2f83-4249-8952-3e660405c723" containerName="revoke-keys" Nov 02 19:07:43.052260 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.052238 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.106275 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.106240 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4"] Nov 02 19:07:43.127394 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.127364 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.127576 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.127410 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.127622 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.127570 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrgg2\" (UniqueName: \"kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.154398 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.154367 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:07:43.154566 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.154409 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:07:43.228989 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.228949 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.229210 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.228998 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.229210 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.229041 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrgg2\" (UniqueName: \"kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.229653 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.229628 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.231456 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.231436 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.286349 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.286318 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrgg2\" (UniqueName: \"kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2\") pod \"maas-api-revoke-sub-ae594979-8kgt4\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.365211 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.365183 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:43.736219 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.736193 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4"] Nov 02 19:07:43.738537 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:07:43.738507 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4aa2c0c_f914_412a_a975_d59159e137aa.slice/crio-9d5866b0037702994d69ad6cbcc6d8fa8de2ec1a1c0c902e6f1c1fd1ed1f8a34 WatchSource:0}: Error finding container 9d5866b0037702994d69ad6cbcc6d8fa8de2ec1a1c0c902e6f1c1fd1ed1f8a34: Status 404 returned error can't find the container with id 9d5866b0037702994d69ad6cbcc6d8fa8de2ec1a1c0c902e6f1c1fd1ed1f8a34 Nov 02 19:07:43.759850 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:43.759803 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" event={"ID":"d4aa2c0c-f914-412a-a975-d59159e137aa","Type":"ContainerStarted","Data":"9d5866b0037702994d69ad6cbcc6d8fa8de2ec1a1c0c902e6f1c1fd1ed1f8a34"} Nov 02 19:07:44.766826 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:44.766786 5647 generic.go:356] "Generic (PLEG): container finished" podID="d4aa2c0c-f914-412a-a975-d59159e137aa" containerID="8d20a48196d2e8d785eba84c3fa5c9f7efc0621ca2dbf5cd82054ea80b7060c0" exitCode=0 Nov 02 19:07:44.767270 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:44.766873 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" event={"ID":"d4aa2c0c-f914-412a-a975-d59159e137aa","Type":"ContainerDied","Data":"8d20a48196d2e8d785eba84c3fa5c9f7efc0621ca2dbf5cd82054ea80b7060c0"} Nov 02 19:07:45.915548 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:45.915523 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:46.057560 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.057468 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca\") pod \"d4aa2c0c-f914-412a-a975-d59159e137aa\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " Nov 02 19:07:46.057560 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.057516 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2\" (UniqueName: \"kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2\") pod \"d4aa2c0c-f914-412a-a975-d59159e137aa\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " Nov 02 19:07:46.057880 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.057595 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca\") pod \"d4aa2c0c-f914-412a-a975-d59159e137aa\" (UID: \"d4aa2c0c-f914-412a-a975-d59159e137aa\") " Nov 02 19:07:46.058009 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.057979 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca" pod "d4aa2c0c-f914-412a-a975-d59159e137aa" (UID: "d4aa2c0c-f914-412a-a975-d59159e137aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:07:46.060542 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.060507 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca" pod "d4aa2c0c-f914-412a-a975-d59159e137aa" (UID: "d4aa2c0c-f914-412a-a975-d59159e137aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:07:46.060717 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.060698 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2" pod "d4aa2c0c-f914-412a-a975-d59159e137aa" (UID: "d4aa2c0c-f914-412a-a975-d59159e137aa"). InnerVolumeSpecName "kube-api-access-wrgg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:07:46.158808 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.158774 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:46.158808 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.158804 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4aa2c0c-f914-412a-a975-d59159e137aa-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:46.158808 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.158815 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrgg2\" (UniqueName: \"kubernetes.io/projected/d4aa2c0c-f914-412a-a975-d59159e137aa-kube-api-access-wrgg2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:07:46.778038 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.778007 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" Nov 02 19:07:46.778254 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.778039 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4" event={"ID":"d4aa2c0c-f914-412a-a975-d59159e137aa","Type":"ContainerDied","Data":"9d5866b0037702994d69ad6cbcc6d8fa8de2ec1a1c0c902e6f1c1fd1ed1f8a34"} Nov 02 19:07:46.778254 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:46.778098 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d5866b0037702994d69ad6cbcc6d8fa8de2ec1a1c0c902e6f1c1fd1ed1f8a34" Nov 02 19:07:55.846380 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:07:55.846346 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:03.152761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:03.150871 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:08:03.152761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:03.150956 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:08:03.486385 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:03.485839 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:08:03.486385 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:03.485924 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:08:04.854022 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.853985 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6"] Nov 02 19:08:04.854491 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.854477 5647 cpu_manager.go:414] "containerMap: removing container" podUID="d4aa2c0c-f914-412a-a975-d59159e137aa" containerName="revoke-keys" Nov 02 19:08:04.854542 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.854493 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4aa2c0c-f914-412a-a975-d59159e137aa" containerName="revoke-keys" Nov 02 19:08:04.854583 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.854573 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4aa2c0c-f914-412a-a975-d59159e137aa" containerName="revoke-keys" Nov 02 19:08:04.858003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.857985 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:04.913576 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.913539 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6"] Nov 02 19:08:04.915228 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.915195 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6x22\" (UniqueName: \"kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:04.915423 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.915319 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:04.915423 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.915364 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:04.962013 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.961981 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:08:04.962194 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:04.962046 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:08:05.016302 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.016264 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6x22\" (UniqueName: \"kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.016481 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.016359 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.016481 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.016412 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.017025 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.017003 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.019022 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.018997 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.073827 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.073790 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6x22\" (UniqueName: \"kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22\") pod \"maas-api-revoke-sub-a4aef7d7-wvzg6\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.171374 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.171287 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:05.268305 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.268268 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:05.542796 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.542759 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6"] Nov 02 19:08:05.544888 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:08:05.544856 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3811f47_bf4d_4960_a4fa_de2ffba7ffec.slice/crio-1f612a10e266d44dc5ab2439cf3d4c90092ce009612b057c9bf00082069fcbb5 WatchSource:0}: Error finding container 1f612a10e266d44dc5ab2439cf3d4c90092ce009612b057c9bf00082069fcbb5: Status 404 returned error can't find the container with id 1f612a10e266d44dc5ab2439cf3d4c90092ce009612b057c9bf00082069fcbb5 Nov 02 19:08:05.920941 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.920896 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" event={"ID":"a3811f47-bf4d-4960-a4fa-de2ffba7ffec","Type":"ContainerStarted","Data":"2c53f87253f783f9d0f58d8553963aa5aa3149d2966eb815888d6d5aa3412960"} Nov 02 19:08:05.921345 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:05.920968 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" event={"ID":"a3811f47-bf4d-4960-a4fa-de2ffba7ffec","Type":"ContainerStarted","Data":"1f612a10e266d44dc5ab2439cf3d4c90092ce009612b057c9bf00082069fcbb5"} Nov 02 19:08:05.947385 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:08:05.947347 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:08:06.035326 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:06.035274 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" podStartSLOduration=2.035260513 podStartE2EDuration="2.035260513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:08:06.034645703 +0000 UTC m=+6273.108172907" watchObservedRunningTime="2026-11-02 19:08:06.035260513 +0000 UTC m=+6273.108787717" Nov 02 19:08:06.929267 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:06.929225 5647 generic.go:356] "Generic (PLEG): container finished" podID="a3811f47-bf4d-4960-a4fa-de2ffba7ffec" containerID="2c53f87253f783f9d0f58d8553963aa5aa3149d2966eb815888d6d5aa3412960" exitCode=0 Nov 02 19:08:06.929759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:06.929323 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" event={"ID":"a3811f47-bf4d-4960-a4fa-de2ffba7ffec","Type":"ContainerDied","Data":"2c53f87253f783f9d0f58d8553963aa5aa3149d2966eb815888d6d5aa3412960"} Nov 02 19:08:08.074536 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.074510 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:08.147890 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.147842 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca\") pod \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " Nov 02 19:08:08.148085 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.147979 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22\" (UniqueName: \"kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22\") pod \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " Nov 02 19:08:08.148085 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.148030 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca\") pod \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\" (UID: \"a3811f47-bf4d-4960-a4fa-de2ffba7ffec\") " Nov 02 19:08:08.148215 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.148194 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca" pod "a3811f47-bf4d-4960-a4fa-de2ffba7ffec" (UID: "a3811f47-bf4d-4960-a4fa-de2ffba7ffec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:08:08.148341 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.148325 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:08.150290 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.150261 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca" pod "a3811f47-bf4d-4960-a4fa-de2ffba7ffec" (UID: "a3811f47-bf4d-4960-a4fa-de2ffba7ffec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:08:08.150398 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.150300 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22" pod "a3811f47-bf4d-4960-a4fa-de2ffba7ffec" (UID: "a3811f47-bf4d-4960-a4fa-de2ffba7ffec"). InnerVolumeSpecName "kube-api-access-b6x22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:08:08.248991 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.248908 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6x22\" (UniqueName: \"kubernetes.io/projected/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-kube-api-access-b6x22\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:08.248991 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.248939 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3811f47-bf4d-4960-a4fa-de2ffba7ffec-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:08.941431 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.941395 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" event={"ID":"a3811f47-bf4d-4960-a4fa-de2ffba7ffec","Type":"ContainerDied","Data":"1f612a10e266d44dc5ab2439cf3d4c90092ce009612b057c9bf00082069fcbb5"} Nov 02 19:08:08.941629 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.941443 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f612a10e266d44dc5ab2439cf3d4c90092ce009612b057c9bf00082069fcbb5" Nov 02 19:08:08.941629 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:08.941404 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6" Nov 02 19:08:13.574964 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:13.574931 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:23.347602 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:23.347556 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:25.967759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:25.967659 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw"] Nov 02 19:08:25.976759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:25.968434 5647 cpu_manager.go:414] "containerMap: removing container" podUID="a3811f47-bf4d-4960-a4fa-de2ffba7ffec" containerName="revoke-keys" Nov 02 19:08:25.976759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:25.968461 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3811f47-bf4d-4960-a4fa-de2ffba7ffec" containerName="revoke-keys" Nov 02 19:08:25.976759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:25.968591 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3811f47-bf4d-4960-a4fa-de2ffba7ffec" containerName="revoke-keys" Nov 02 19:08:25.986147 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:25.986115 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.048802 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.044145 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw"] Nov 02 19:08:26.147412 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.147362 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.147758 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.147711 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.148021 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.148001 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nll5\" (UniqueName: \"kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.148480 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.148457 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:08:26.148685 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.148668 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:08:26.290834 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.285946 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.290834 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.286019 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.290834 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.286122 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nll5\" (UniqueName: \"kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.302873 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.301869 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.324903 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.324869 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.427763 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.411646 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nll5\" (UniqueName: \"kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5\") pod \"maas-api-revoke-sub-79805ca9-7s5dw\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:26.654216 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:26.653915 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:27.123489 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:27.123466 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw"] Nov 02 19:08:27.125876 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:08:27.125850 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c27f38_2846_456a_b13e_2d2d81bcde32.slice/crio-d21411f8d16cfd559781a68dde2cf6431af18d10dff6c1be922f9eacf21b29f2 WatchSource:0}: Error finding container d21411f8d16cfd559781a68dde2cf6431af18d10dff6c1be922f9eacf21b29f2: Status 404 returned error can't find the container with id d21411f8d16cfd559781a68dde2cf6431af18d10dff6c1be922f9eacf21b29f2 Nov 02 19:08:27.298076 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:27.298032 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" event={"ID":"38c27f38-2846-456a-b13e-2d2d81bcde32","Type":"ContainerStarted","Data":"d21411f8d16cfd559781a68dde2cf6431af18d10dff6c1be922f9eacf21b29f2"} Nov 02 19:08:28.304785 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:28.304671 5647 generic.go:356] "Generic (PLEG): container finished" podID="38c27f38-2846-456a-b13e-2d2d81bcde32" containerID="4fc91e61bb21889d39eccb004cda5722db96ec54549a15a4e236c89d7a09aa30" exitCode=0 Nov 02 19:08:28.304785 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:28.304761 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" event={"ID":"38c27f38-2846-456a-b13e-2d2d81bcde32","Type":"ContainerDied","Data":"4fc91e61bb21889d39eccb004cda5722db96ec54549a15a4e236c89d7a09aa30"} Nov 02 19:08:29.451549 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.451524 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:29.561186 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.561109 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca\") pod \"38c27f38-2846-456a-b13e-2d2d81bcde32\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " Nov 02 19:08:29.561365 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.561200 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5\" (UniqueName: \"kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5\") pod \"38c27f38-2846-456a-b13e-2d2d81bcde32\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " Nov 02 19:08:29.561365 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.561259 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca\") pod \"38c27f38-2846-456a-b13e-2d2d81bcde32\" (UID: \"38c27f38-2846-456a-b13e-2d2d81bcde32\") " Nov 02 19:08:29.561695 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.561671 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca" pod "38c27f38-2846-456a-b13e-2d2d81bcde32" (UID: "38c27f38-2846-456a-b13e-2d2d81bcde32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:08:29.563420 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.563372 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca" pod "38c27f38-2846-456a-b13e-2d2d81bcde32" (UID: "38c27f38-2846-456a-b13e-2d2d81bcde32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:08:29.563625 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.563601 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5" pod "38c27f38-2846-456a-b13e-2d2d81bcde32" (UID: "38c27f38-2846-456a-b13e-2d2d81bcde32"). InnerVolumeSpecName "kube-api-access-2nll5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:08:29.662210 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.662174 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nll5\" (UniqueName: \"kubernetes.io/projected/38c27f38-2846-456a-b13e-2d2d81bcde32-kube-api-access-2nll5\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:29.662210 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.662207 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:29.662210 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:29.662218 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38c27f38-2846-456a-b13e-2d2d81bcde32-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:30.316339 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:30.316311 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" Nov 02 19:08:30.316339 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:30.316318 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw" event={"ID":"38c27f38-2846-456a-b13e-2d2d81bcde32","Type":"ContainerDied","Data":"d21411f8d16cfd559781a68dde2cf6431af18d10dff6c1be922f9eacf21b29f2"} Nov 02 19:08:30.316584 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:30.316369 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d21411f8d16cfd559781a68dde2cf6431af18d10dff6c1be922f9eacf21b29f2" Nov 02 19:08:32.572926 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:32.572890 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:35.085193 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:35.085157 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:08:35.093282 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:35.093256 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:08:35.946914 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:08:35.946868 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:08:38.456407 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:38.456375 5647 scope.go:122] "RemoveContainer" containerID="83a6482ff3c6c3ff6e0e6304a3be8c8606a8d361aad14034378d571d6c3bb93b" Nov 02 19:08:38.465665 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:38.465644 5647 scope.go:122] "RemoveContainer" containerID="36af82c586b0869992deef437eb218d7d2789ff46872470f039ecce909c9a198" Nov 02 19:08:38.473644 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:38.473623 5647 scope.go:122] "RemoveContainer" containerID="7da2c21fd3df243ef37e0d720bffa1c632a1d70a176657d9a601f0aff02ff78e" Nov 02 19:08:41.464673 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:41.464578 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:43.422761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.421323 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph"] Nov 02 19:08:43.422761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.422068 5647 cpu_manager.go:414] "containerMap: removing container" podUID="38c27f38-2846-456a-b13e-2d2d81bcde32" containerName="revoke-keys" Nov 02 19:08:43.422761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.422088 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c27f38-2846-456a-b13e-2d2d81bcde32" containerName="revoke-keys" Nov 02 19:08:43.422761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.422209 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c27f38-2846-456a-b13e-2d2d81bcde32" containerName="revoke-keys" Nov 02 19:08:43.435758 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.428658 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.491769 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.490429 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph"] Nov 02 19:08:43.582923 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.582879 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:08:43.584534 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.582893 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:08:43.584761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.584381 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqx2r\" (UniqueName: \"kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.585058 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.585021 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.585269 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.585235 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.694026 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.693890 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqx2r\" (UniqueName: \"kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.694900 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.694864 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.695939 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.695917 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.696918 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.696896 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.745782 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.738056 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.806990 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.801458 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqx2r\" (UniqueName: \"kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r\") pod \"maas-api-revoke-sub-0b7e8edb-85qph\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:43.889080 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:43.889034 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:44.621491 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:44.621449 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph"] Nov 02 19:08:45.598643 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:45.598605 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" event={"ID":"77630e73-3f8d-4bed-bef7-740154add03f","Type":"ContainerStarted","Data":"8b560aa42472cf566f80fb7aefd279dd1bf5abb7e1c5873f9708b577bc4b2ec3"} Nov 02 19:08:45.598883 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:45.598661 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" event={"ID":"77630e73-3f8d-4bed-bef7-740154add03f","Type":"ContainerStarted","Data":"0e6085f0f2f6ec51275b75952391392dd21633f2559301a79f1e56ab2b8a7f16"} Nov 02 19:08:45.714282 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:45.714212 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" podStartSLOduration=2.714196737 podStartE2EDuration="2.714196737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:08:45.712570638 +0000 UTC m=+6312.786097843" watchObservedRunningTime="2026-11-02 19:08:45.714196737 +0000 UTC m=+6312.787723940" Nov 02 19:08:46.606660 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:46.606626 5647 generic.go:356] "Generic (PLEG): container finished" podID="77630e73-3f8d-4bed-bef7-740154add03f" containerID="8b560aa42472cf566f80fb7aefd279dd1bf5abb7e1c5873f9708b577bc4b2ec3" exitCode=0 Nov 02 19:08:46.606859 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:46.606706 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" event={"ID":"77630e73-3f8d-4bed-bef7-740154add03f","Type":"ContainerDied","Data":"8b560aa42472cf566f80fb7aefd279dd1bf5abb7e1c5873f9708b577bc4b2ec3"} Nov 02 19:08:47.753627 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.753602 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:47.766821 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.766797 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r\" (UniqueName: \"kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r\") pod \"77630e73-3f8d-4bed-bef7-740154add03f\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " Nov 02 19:08:47.766922 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.766858 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca\") pod \"77630e73-3f8d-4bed-bef7-740154add03f\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " Nov 02 19:08:47.766922 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.766915 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca\") pod \"77630e73-3f8d-4bed-bef7-740154add03f\" (UID: \"77630e73-3f8d-4bed-bef7-740154add03f\") " Nov 02 19:08:47.767295 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.767273 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca" pod "77630e73-3f8d-4bed-bef7-740154add03f" (UID: "77630e73-3f8d-4bed-bef7-740154add03f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:08:47.769155 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.769133 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r" pod "77630e73-3f8d-4bed-bef7-740154add03f" (UID: "77630e73-3f8d-4bed-bef7-740154add03f"). InnerVolumeSpecName "kube-api-access-bqx2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:08:47.769252 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.769150 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca" pod "77630e73-3f8d-4bed-bef7-740154add03f" (UID: "77630e73-3f8d-4bed-bef7-740154add03f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:08:47.868253 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.868124 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:47.868253 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.868159 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqx2r\" (UniqueName: \"kubernetes.io/projected/77630e73-3f8d-4bed-bef7-740154add03f-kube-api-access-bqx2r\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:47.868253 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:47.868175 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77630e73-3f8d-4bed-bef7-740154add03f-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:48.620955 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:48.620916 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" event={"ID":"77630e73-3f8d-4bed-bef7-740154add03f","Type":"ContainerDied","Data":"0e6085f0f2f6ec51275b75952391392dd21633f2559301a79f1e56ab2b8a7f16"} Nov 02 19:08:48.620955 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:48.620966 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6085f0f2f6ec51275b75952391392dd21633f2559301a79f1e56ab2b8a7f16" Nov 02 19:08:48.621281 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:48.621017 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph" Nov 02 19:08:49.158891 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:49.158854 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:08:54.721094 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.721055 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz"] Nov 02 19:08:54.721619 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.721552 5647 cpu_manager.go:414] "containerMap: removing container" podUID="77630e73-3f8d-4bed-bef7-740154add03f" containerName="revoke-keys" Nov 02 19:08:54.721619 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.721565 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77630e73-3f8d-4bed-bef7-740154add03f" containerName="revoke-keys" Nov 02 19:08:54.721721 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.721641 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="77630e73-3f8d-4bed-bef7-740154add03f" containerName="revoke-keys" Nov 02 19:08:54.724481 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.724465 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.828546 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.828513 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:08:54.828546 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.828513 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:08:54.830608 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.830583 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz"] Nov 02 19:08:54.850713 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.850672 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.850905 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.850719 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkrvc\" (UniqueName: \"kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.850905 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.850819 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.952090 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.952047 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.952090 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.952094 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkrvc\" (UniqueName: \"kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.952378 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.952222 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.952716 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.952689 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:54.954907 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:54.954886 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:55.010410 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:55.010339 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkrvc\" (UniqueName: \"kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc\") pod \"maas-api-revoke-sub-3d4156f1-b45wz\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:55.038196 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:55.038157 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:55.409434 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:55.409409 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz"] Nov 02 19:08:55.411370 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:08:55.411341 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12d7887_7c57_43a6_b983_1c90efffa2b8.slice/crio-10e8d3ca57bd280139ac1048dbec3e3a18457d72f88f98c1c9df5a3a9369973b WatchSource:0}: Error finding container 10e8d3ca57bd280139ac1048dbec3e3a18457d72f88f98c1c9df5a3a9369973b: Status 404 returned error can't find the container with id 10e8d3ca57bd280139ac1048dbec3e3a18457d72f88f98c1c9df5a3a9369973b Nov 02 19:08:55.660375 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:55.660294 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" event={"ID":"f12d7887-7c57-43a6-b983-1c90efffa2b8","Type":"ContainerStarted","Data":"10e8d3ca57bd280139ac1048dbec3e3a18457d72f88f98c1c9df5a3a9369973b"} Nov 02 19:08:56.667557 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:56.667521 5647 generic.go:356] "Generic (PLEG): container finished" podID="f12d7887-7c57-43a6-b983-1c90efffa2b8" containerID="e99f84f1ad530f13b5f59a73a3832dbd88a48a6d41099a12f6c27d4ff2f69595" exitCode=0 Nov 02 19:08:56.668006 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:56.667605 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" event={"ID":"f12d7887-7c57-43a6-b983-1c90efffa2b8","Type":"ContainerDied","Data":"e99f84f1ad530f13b5f59a73a3832dbd88a48a6d41099a12f6c27d4ff2f69595"} Nov 02 19:08:57.810478 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.810446 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:57.883678 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.883647 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca\") pod \"f12d7887-7c57-43a6-b983-1c90efffa2b8\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " Nov 02 19:08:57.883870 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.883766 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc\" (UniqueName: \"kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc\") pod \"f12d7887-7c57-43a6-b983-1c90efffa2b8\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " Nov 02 19:08:57.883870 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.883790 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca\") pod \"f12d7887-7c57-43a6-b983-1c90efffa2b8\" (UID: \"f12d7887-7c57-43a6-b983-1c90efffa2b8\") " Nov 02 19:08:57.884246 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.884221 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca" pod "f12d7887-7c57-43a6-b983-1c90efffa2b8" (UID: "f12d7887-7c57-43a6-b983-1c90efffa2b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:08:57.885916 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.885892 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca" pod "f12d7887-7c57-43a6-b983-1c90efffa2b8" (UID: "f12d7887-7c57-43a6-b983-1c90efffa2b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:08:57.886006 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.885911 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc" pod "f12d7887-7c57-43a6-b983-1c90efffa2b8" (UID: "f12d7887-7c57-43a6-b983-1c90efffa2b8"). InnerVolumeSpecName "kube-api-access-lkrvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:08:57.985491 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.985389 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:57.985491 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.985438 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkrvc\" (UniqueName: \"kubernetes.io/projected/f12d7887-7c57-43a6-b983-1c90efffa2b8-kube-api-access-lkrvc\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:57.985491 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:57.985448 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12d7887-7c57-43a6-b983-1c90efffa2b8-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:08:58.678429 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:58.678390 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" event={"ID":"f12d7887-7c57-43a6-b983-1c90efffa2b8","Type":"ContainerDied","Data":"10e8d3ca57bd280139ac1048dbec3e3a18457d72f88f98c1c9df5a3a9369973b"} Nov 02 19:08:58.678429 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:58.678420 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz" Nov 02 19:08:58.678656 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:08:58.678449 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e8d3ca57bd280139ac1048dbec3e3a18457d72f88f98c1c9df5a3a9369973b" Nov 02 19:09:05.946207 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:09:05.946167 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:09:09.248168 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:09.248122 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:09:18.135760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:18.133858 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:09:18.135760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:18.133935 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:09:18.496972 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:18.496856 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:09:18.496972 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:18.496928 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:09:18.982586 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:18.982551 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz"] Nov 02 19:09:19.033253 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.033215 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b45wz"] Nov 02 19:09:19.080931 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.080896 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12d7887-7c57-43a6-b983-1c90efffa2b8" path="/var/lib/kubelet/pods/f12d7887-7c57-43a6-b983-1c90efffa2b8/volumes" Nov 02 19:09:19.101226 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.101193 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2"] Nov 02 19:09:19.101653 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.101641 5647 cpu_manager.go:414] "containerMap: removing container" podUID="f12d7887-7c57-43a6-b983-1c90efffa2b8" containerName="revoke-keys" Nov 02 19:09:19.101703 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.101655 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12d7887-7c57-43a6-b983-1c90efffa2b8" containerName="revoke-keys" Nov 02 19:09:19.101757 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.101751 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12d7887-7c57-43a6-b983-1c90efffa2b8" containerName="revoke-keys" Nov 02 19:09:19.105124 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.105093 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.159356 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.159262 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:09:19.159356 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.159329 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:09:19.160831 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.160803 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2"] Nov 02 19:09:19.190946 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.190914 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.191138 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.190983 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrlg\" (UniqueName: \"kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.191138 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.191087 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.292309 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.292211 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.292309 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.292288 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrlg\" (UniqueName: \"kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.292528 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.292320 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.293023 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.293001 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.294960 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.294940 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.350191 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.350160 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrlg\" (UniqueName: \"kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg\") pod \"maas-api-revoke-sub-3d4156f1-cl9z2\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.418701 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.418664 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:19.795084 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.795054 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2"] Nov 02 19:09:19.796201 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:09:19.796166 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cd876e_4230_41e6_a706_c44c17e7702c.slice/crio-8662e28df1176f90508e1a1cb52adcf67c376a0cd0eb7805110a3fc43600e035 WatchSource:0}: Error finding container 8662e28df1176f90508e1a1cb52adcf67c376a0cd0eb7805110a3fc43600e035: Status 404 returned error can't find the container with id 8662e28df1176f90508e1a1cb52adcf67c376a0cd0eb7805110a3fc43600e035 Nov 02 19:09:19.861103 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:19.861069 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" event={"ID":"81cd876e-4230-41e6-a706-c44c17e7702c","Type":"ContainerStarted","Data":"8662e28df1176f90508e1a1cb52adcf67c376a0cd0eb7805110a3fc43600e035"} Nov 02 19:09:20.130872 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:20.130845 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:09:20.867942 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:20.867899 5647 generic.go:356] "Generic (PLEG): container finished" podID="81cd876e-4230-41e6-a706-c44c17e7702c" containerID="e42bbadd637eff07ff7ee02658935b381b80310f5ea001989537528cd0e716cd" exitCode=0 Nov 02 19:09:20.868337 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:20.867956 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" event={"ID":"81cd876e-4230-41e6-a706-c44c17e7702c","Type":"ContainerDied","Data":"e42bbadd637eff07ff7ee02658935b381b80310f5ea001989537528cd0e716cd"} Nov 02 19:09:22.006197 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.006173 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:22.120091 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.120055 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca\") pod \"81cd876e-4230-41e6-a706-c44c17e7702c\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " Nov 02 19:09:22.120347 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.120140 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca\") pod \"81cd876e-4230-41e6-a706-c44c17e7702c\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " Nov 02 19:09:22.120347 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.120193 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg\" (UniqueName: \"kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg\") pod \"81cd876e-4230-41e6-a706-c44c17e7702c\" (UID: \"81cd876e-4230-41e6-a706-c44c17e7702c\") " Nov 02 19:09:22.120647 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.120628 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca" pod "81cd876e-4230-41e6-a706-c44c17e7702c" (UID: "81cd876e-4230-41e6-a706-c44c17e7702c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:09:22.122938 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.122907 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca" pod "81cd876e-4230-41e6-a706-c44c17e7702c" (UID: "81cd876e-4230-41e6-a706-c44c17e7702c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:09:22.123095 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.123009 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg" pod "81cd876e-4230-41e6-a706-c44c17e7702c" (UID: "81cd876e-4230-41e6-a706-c44c17e7702c"). InnerVolumeSpecName "kube-api-access-xtrlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:09:22.221483 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.221454 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:09:22.221483 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.221481 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cd876e-4230-41e6-a706-c44c17e7702c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:09:22.221483 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.221491 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtrlg\" (UniqueName: \"kubernetes.io/projected/81cd876e-4230-41e6-a706-c44c17e7702c-kube-api-access-xtrlg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:09:22.880470 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.880439 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" Nov 02 19:09:22.880470 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.880466 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2" event={"ID":"81cd876e-4230-41e6-a706-c44c17e7702c","Type":"ContainerDied","Data":"8662e28df1176f90508e1a1cb52adcf67c376a0cd0eb7805110a3fc43600e035"} Nov 02 19:09:22.880706 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:22.880507 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8662e28df1176f90508e1a1cb52adcf67c376a0cd0eb7805110a3fc43600e035" Nov 02 19:09:33.551601 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:33.551556 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:09:35.946800 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:09:35.946762 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:09:43.446145 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:43.446108 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:09:52.609917 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:52.609876 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:09:52.610607 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:52.610569 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:09:55.027659 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:09:55.027619 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:10:04.392151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.392113 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd"] Nov 02 19:10:04.392709 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.392564 5647 cpu_manager.go:414] "containerMap: removing container" podUID="81cd876e-4230-41e6-a706-c44c17e7702c" containerName="revoke-keys" Nov 02 19:10:04.392709 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.392575 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cd876e-4230-41e6-a706-c44c17e7702c" containerName="revoke-keys" Nov 02 19:10:04.392709 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.392644 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cd876e-4230-41e6-a706-c44c17e7702c" containerName="revoke-keys" Nov 02 19:10:04.395896 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.395877 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.452229 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.452194 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd"] Nov 02 19:10:04.500057 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.500024 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:10:04.500057 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.500068 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:10:04.513084 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.513051 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbtjd\" (UniqueName: \"kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.513254 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.513098 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.513356 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.513331 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.614995 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.614960 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.615188 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.615026 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbtjd\" (UniqueName: \"kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.615188 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.615059 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.615707 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.615687 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.617433 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.617415 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.672168 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.672095 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbtjd\" (UniqueName: \"kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd\") pod \"maas-api-revoke-sub-e2fc9c5b-9m9qd\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:04.709060 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:04.709022 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:05.079497 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:10:05.079465 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0225158d_2be3_4485_9097_ab53db834898.slice/crio-8fffaee899398470eaea57c9552b84ec834c72a50df33d8b360bae083f24ce64 WatchSource:0}: Error finding container 8fffaee899398470eaea57c9552b84ec834c72a50df33d8b360bae083f24ce64: Status 404 returned error can't find the container with id 8fffaee899398470eaea57c9552b84ec834c72a50df33d8b360bae083f24ce64 Nov 02 19:10:05.082010 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:05.081991 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd"] Nov 02 19:10:05.130954 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:05.130917 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" event={"ID":"0225158d-2be3-4485-9097-ab53db834898","Type":"ContainerStarted","Data":"8fffaee899398470eaea57c9552b84ec834c72a50df33d8b360bae083f24ce64"} Nov 02 19:10:05.946681 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:10:05.946642 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:10:06.137451 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:06.137409 5647 generic.go:356] "Generic (PLEG): container finished" podID="0225158d-2be3-4485-9097-ab53db834898" containerID="37a11868aa7705a26839e0a3412113393e05028df23f31e6d5704ded1efe09fd" exitCode=0 Nov 02 19:10:06.137649 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:06.137493 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" event={"ID":"0225158d-2be3-4485-9097-ab53db834898","Type":"ContainerDied","Data":"37a11868aa7705a26839e0a3412113393e05028df23f31e6d5704ded1efe09fd"} Nov 02 19:10:07.278625 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.278600 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:07.441421 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.441330 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca\") pod \"0225158d-2be3-4485-9097-ab53db834898\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " Nov 02 19:10:07.441421 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.441396 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd\" (UniqueName: \"kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd\") pod \"0225158d-2be3-4485-9097-ab53db834898\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " Nov 02 19:10:07.441668 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.441524 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca\") pod \"0225158d-2be3-4485-9097-ab53db834898\" (UID: \"0225158d-2be3-4485-9097-ab53db834898\") " Nov 02 19:10:07.441946 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.441924 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca" pod "0225158d-2be3-4485-9097-ab53db834898" (UID: "0225158d-2be3-4485-9097-ab53db834898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:10:07.443684 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.443665 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd" pod "0225158d-2be3-4485-9097-ab53db834898" (UID: "0225158d-2be3-4485-9097-ab53db834898"). InnerVolumeSpecName "kube-api-access-bbtjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:10:07.443778 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.443750 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca" pod "0225158d-2be3-4485-9097-ab53db834898" (UID: "0225158d-2be3-4485-9097-ab53db834898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:10:07.542656 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.542624 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0225158d-2be3-4485-9097-ab53db834898-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:07.542656 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.542652 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0225158d-2be3-4485-9097-ab53db834898-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:07.542656 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:07.542662 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbtjd\" (UniqueName: \"kubernetes.io/projected/0225158d-2be3-4485-9097-ab53db834898-kube-api-access-bbtjd\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:08.149967 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:08.149933 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" Nov 02 19:10:08.150155 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:08.149969 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd" event={"ID":"0225158d-2be3-4485-9097-ab53db834898","Type":"ContainerDied","Data":"8fffaee899398470eaea57c9552b84ec834c72a50df33d8b360bae083f24ce64"} Nov 02 19:10:08.150155 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:08.150021 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fffaee899398470eaea57c9552b84ec834c72a50df33d8b360bae083f24ce64" Nov 02 19:10:15.135711 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.135610 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw"] Nov 02 19:10:15.136337 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.136314 5647 cpu_manager.go:414] "containerMap: removing container" podUID="0225158d-2be3-4485-9097-ab53db834898" containerName="revoke-keys" Nov 02 19:10:15.136433 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.136339 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0225158d-2be3-4485-9097-ab53db834898" containerName="revoke-keys" Nov 02 19:10:15.136499 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.136463 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="0225158d-2be3-4485-9097-ab53db834898" containerName="revoke-keys" Nov 02 19:10:15.140105 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.140076 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.195343 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.195308 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw"] Nov 02 19:10:15.213073 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.213037 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.213267 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.213162 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.213267 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.213207 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpvjg\" (UniqueName: \"kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.243427 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.243386 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:10:15.243630 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.243438 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:10:15.313724 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.313688 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.313970 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.313810 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.313970 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.313858 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpvjg\" (UniqueName: \"kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.314536 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.314505 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.316351 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.316326 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.372239 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.372203 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpvjg\" (UniqueName: \"kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg\") pod \"maas-api-revoke-sub-5a29efb3-d6kcw\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.453998 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.453908 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:15.824533 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:15.824507 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw"] Nov 02 19:10:15.825865 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:10:15.825840 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4730a3_5299_4f38_98c6_544bf7c7d47f.slice/crio-cd2d5b1e2d7b6dcfedc593facf658c037630a89b17a5912e0ac03524fc6243c2 WatchSource:0}: Error finding container cd2d5b1e2d7b6dcfedc593facf658c037630a89b17a5912e0ac03524fc6243c2: Status 404 returned error can't find the container with id cd2d5b1e2d7b6dcfedc593facf658c037630a89b17a5912e0ac03524fc6243c2 Nov 02 19:10:16.193526 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:16.193490 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" event={"ID":"ba4730a3-5299-4f38-98c6-544bf7c7d47f","Type":"ContainerStarted","Data":"8f8bca51abb646d2ecdb95fd8518efc7cea76eded26db890a2fe2bef02c665fc"} Nov 02 19:10:16.194005 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:16.193546 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" event={"ID":"ba4730a3-5299-4f38-98c6-544bf7c7d47f","Type":"ContainerStarted","Data":"cd2d5b1e2d7b6dcfedc593facf658c037630a89b17a5912e0ac03524fc6243c2"} Nov 02 19:10:16.309131 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:16.309075 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" podStartSLOduration=1.30906069 podStartE2EDuration="1.30906069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:10:16.307112725 +0000 UTC m=+6403.380639931" watchObservedRunningTime="2026-11-02 19:10:16.30906069 +0000 UTC m=+6403.382587892" Nov 02 19:10:17.200495 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:17.200459 5647 generic.go:356] "Generic (PLEG): container finished" podID="ba4730a3-5299-4f38-98c6-544bf7c7d47f" containerID="8f8bca51abb646d2ecdb95fd8518efc7cea76eded26db890a2fe2bef02c665fc" exitCode=0 Nov 02 19:10:17.200918 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:17.200537 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" event={"ID":"ba4730a3-5299-4f38-98c6-544bf7c7d47f","Type":"ContainerDied","Data":"8f8bca51abb646d2ecdb95fd8518efc7cea76eded26db890a2fe2bef02c665fc"} Nov 02 19:10:17.252794 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:17.252761 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:10:18.045419 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.045375 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9"] Nov 02 19:10:18.095896 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.095849 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-wg5m9"] Nov 02 19:10:18.406048 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.406011 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:18.448471 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.448410 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca\") pod \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " Nov 02 19:10:18.448694 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.448488 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg\" (UniqueName: \"kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg\") pod \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " Nov 02 19:10:18.448694 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.448545 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca\") pod \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\" (UID: \"ba4730a3-5299-4f38-98c6-544bf7c7d47f\") " Nov 02 19:10:18.449270 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.449233 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca" pod "ba4730a3-5299-4f38-98c6-544bf7c7d47f" (UID: "ba4730a3-5299-4f38-98c6-544bf7c7d47f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:10:18.451550 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.451519 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca" pod "ba4730a3-5299-4f38-98c6-544bf7c7d47f" (UID: "ba4730a3-5299-4f38-98c6-544bf7c7d47f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:10:18.452170 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.452141 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg" pod "ba4730a3-5299-4f38-98c6-544bf7c7d47f" (UID: "ba4730a3-5299-4f38-98c6-544bf7c7d47f"). InnerVolumeSpecName "kube-api-access-bpvjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:10:18.550502 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.550448 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:18.550502 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.550507 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpvjg\" (UniqueName: \"kubernetes.io/projected/ba4730a3-5299-4f38-98c6-544bf7c7d47f-kube-api-access-bpvjg\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:18.550855 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:18.550526 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba4730a3-5299-4f38-98c6-544bf7c7d47f-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:19.092893 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:19.092830 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9ab9c2-f452-4e93-b43f-26800dcd2625" path="/var/lib/kubelet/pods/0f9ab9c2-f452-4e93-b43f-26800dcd2625/volumes" Nov 02 19:10:19.260762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:19.258480 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" event={"ID":"ba4730a3-5299-4f38-98c6-544bf7c7d47f","Type":"ContainerDied","Data":"cd2d5b1e2d7b6dcfedc593facf658c037630a89b17a5912e0ac03524fc6243c2"} Nov 02 19:10:19.260762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:19.258569 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2d5b1e2d7b6dcfedc593facf658c037630a89b17a5912e0ac03524fc6243c2" Nov 02 19:10:19.260762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:19.258690 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw" Nov 02 19:10:27.951071 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:27.951033 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:10:34.339439 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.339319 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2"] Nov 02 19:10:34.342665 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.342635 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cl9z2"] Nov 02 19:10:34.457777 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.457743 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb"] Nov 02 19:10:34.458253 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.458238 5647 cpu_manager.go:414] "containerMap: removing container" podUID="ba4730a3-5299-4f38-98c6-544bf7c7d47f" containerName="revoke-keys" Nov 02 19:10:34.458300 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.458255 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4730a3-5299-4f38-98c6-544bf7c7d47f" containerName="revoke-keys" Nov 02 19:10:34.458349 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.458341 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4730a3-5299-4f38-98c6-544bf7c7d47f" containerName="revoke-keys" Nov 02 19:10:34.461544 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.461529 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.515305 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.515264 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:10:34.515564 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.515536 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:10:34.517566 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.517540 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb"] Nov 02 19:10:34.571345 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.571305 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.571567 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.571356 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4rr\" (UniqueName: \"kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.571636 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.571557 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.673151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.673050 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.673151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.673112 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.673422 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.673155 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4rr\" (UniqueName: \"kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.673839 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.673815 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.675586 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.675564 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.730530 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.730502 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4rr\" (UniqueName: \"kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr\") pod \"maas-api-revoke-sub-3d4156f1-fmdkb\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:34.774491 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:34.774455 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:35.080251 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:35.080166 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cd876e-4230-41e6-a706-c44c17e7702c" path="/var/lib/kubelet/pods/81cd876e-4230-41e6-a706-c44c17e7702c/volumes" Nov 02 19:10:35.144197 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:35.144168 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb"] Nov 02 19:10:35.146420 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:10:35.146381 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35e905ab_1bcd_46ea_b24c_5b606dcb9053.slice/crio-73ff6c8e134b9ae905d4b6366688399fe801366eaab19e1d7225d2e0266e4494 WatchSource:0}: Error finding container 73ff6c8e134b9ae905d4b6366688399fe801366eaab19e1d7225d2e0266e4494: Status 404 returned error can't find the container with id 73ff6c8e134b9ae905d4b6366688399fe801366eaab19e1d7225d2e0266e4494 Nov 02 19:10:35.384803 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:35.384771 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" event={"ID":"35e905ab-1bcd-46ea-b24c-5b606dcb9053","Type":"ContainerStarted","Data":"73ff6c8e134b9ae905d4b6366688399fe801366eaab19e1d7225d2e0266e4494"} Nov 02 19:10:35.946534 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:10:35.946497 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:10:36.391832 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:36.391794 5647 generic.go:356] "Generic (PLEG): container finished" podID="35e905ab-1bcd-46ea-b24c-5b606dcb9053" containerID="790296bb6fe0afaff780b73663058290880526ed6e7f7dd1c3f91f59966eed37" exitCode=0 Nov 02 19:10:36.392288 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:36.391877 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" event={"ID":"35e905ab-1bcd-46ea-b24c-5b606dcb9053","Type":"ContainerDied","Data":"790296bb6fe0afaff780b73663058290880526ed6e7f7dd1c3f91f59966eed37"} Nov 02 19:10:37.529551 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.529524 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:37.602795 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.602758 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca\") pod \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " Nov 02 19:10:37.603004 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.602825 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca\") pod \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " Nov 02 19:10:37.603004 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.602988 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr\" (UniqueName: \"kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr\") pod \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\" (UID: \"35e905ab-1bcd-46ea-b24c-5b606dcb9053\") " Nov 02 19:10:37.603182 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.603162 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca" pod "35e905ab-1bcd-46ea-b24c-5b606dcb9053" (UID: "35e905ab-1bcd-46ea-b24c-5b606dcb9053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:10:37.603350 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.603335 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:37.605002 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.604980 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca" pod "35e905ab-1bcd-46ea-b24c-5b606dcb9053" (UID: "35e905ab-1bcd-46ea-b24c-5b606dcb9053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:10:37.605075 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.605022 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr" pod "35e905ab-1bcd-46ea-b24c-5b606dcb9053" (UID: "35e905ab-1bcd-46ea-b24c-5b606dcb9053"). InnerVolumeSpecName "kube-api-access-8f4rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:10:37.704942 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.704863 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f4rr\" (UniqueName: \"kubernetes.io/projected/35e905ab-1bcd-46ea-b24c-5b606dcb9053-kube-api-access-8f4rr\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:37.704942 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:37.704894 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35e905ab-1bcd-46ea-b24c-5b606dcb9053-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:10:38.404074 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:38.404034 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" event={"ID":"35e905ab-1bcd-46ea-b24c-5b606dcb9053","Type":"ContainerDied","Data":"73ff6c8e134b9ae905d4b6366688399fe801366eaab19e1d7225d2e0266e4494"} Nov 02 19:10:38.404261 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:38.404095 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ff6c8e134b9ae905d4b6366688399fe801366eaab19e1d7225d2e0266e4494" Nov 02 19:10:38.404261 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:38.404049 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb" Nov 02 19:10:49.039107 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:49.039064 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:10:49.091206 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:49.091172 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b"] Nov 02 19:10:49.094041 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:49.094014 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-mzg6b"] Nov 02 19:10:51.101677 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:10:51.101639 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de19e992-faa2-4a8f-8f2c-815c2258766c" path="/var/lib/kubelet/pods/de19e992-faa2-4a8f-8f2c-815c2258766c/volumes" Nov 02 19:11:00.840878 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:00.840842 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:11:01.164717 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.164619 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r"] Nov 02 19:11:01.165177 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.165159 5647 cpu_manager.go:414] "containerMap: removing container" podUID="35e905ab-1bcd-46ea-b24c-5b606dcb9053" containerName="revoke-keys" Nov 02 19:11:01.165240 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.165180 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35e905ab-1bcd-46ea-b24c-5b606dcb9053" containerName="revoke-keys" Nov 02 19:11:01.165281 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.165248 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="35e905ab-1bcd-46ea-b24c-5b606dcb9053" containerName="revoke-keys" Nov 02 19:11:01.168436 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.168420 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.179466 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.179444 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.179595 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.179506 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnslk\" (UniqueName: \"kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.179654 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.179607 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.179695 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.179656 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.274791 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.274727 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-5b22de-serving-cert\"" Nov 02 19:11:01.274791 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.274764 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 19:11:01.275034 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.274804 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94n2w\"" Nov 02 19:11:01.277077 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.277055 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r"] Nov 02 19:11:01.280391 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.280373 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnslk\" (UniqueName: \"kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.280443 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.280416 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.280489 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.280451 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.280544 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.280505 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.280643 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:11:01.280622 5647 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-serving-cert: secret "maas-api-e2e-worker-single-main-5b22de-serving-cert" not found Nov 02 19:11:01.280710 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:11:01.280703 5647 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls podName:244bbfd0-a739-42b8-a3bb-e225bc49a43e nodeName:}" failed. No retries permitted until 2026-11-02 19:11:01.780680308 +0000 UTC m=+6448.854207503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls") pod "maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" (UID: "244bbfd0-a739-42b8-a3bb-e225bc49a43e") : secret "maas-api-e2e-worker-single-main-5b22de-serving-cert" not found Nov 02 19:11:01.281213 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.281195 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.283019 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.282994 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.341032 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.341000 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnslk\" (UniqueName: \"kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.790659 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.790620 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:01.793265 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:01.793242 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:02.081589 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:02.081490 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:02.456585 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:02.456557 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r"] Nov 02 19:11:02.457906 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:11:02.457876 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244bbfd0_a739_42b8_a3bb_e225bc49a43e.slice/crio-25a0bc72f574e06476a527416a44b1c4b40c1b1c3999e21271667191cde8cd71 WatchSource:0}: Error finding container 25a0bc72f574e06476a527416a44b1c4b40c1b1c3999e21271667191cde8cd71: Status 404 returned error can't find the container with id 25a0bc72f574e06476a527416a44b1c4b40c1b1c3999e21271667191cde8cd71 Nov 02 19:11:02.459278 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:02.459260 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:11:02.569857 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:02.569815 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" event={"ID":"244bbfd0-a739-42b8-a3bb-e225bc49a43e","Type":"ContainerStarted","Data":"25a0bc72f574e06476a527416a44b1c4b40c1b1c3999e21271667191cde8cd71"} Nov 02 19:11:03.576798 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:03.576756 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" event={"ID":"244bbfd0-a739-42b8-a3bb-e225bc49a43e","Type":"ContainerStarted","Data":"e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38"} Nov 02 19:11:03.577245 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:03.576838 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:03.693916 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:03.693813 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" podStartSLOduration=1.80244005 podStartE2EDuration="2.693797804s" totalImagesPullingTime="891.357754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:11:03.693401914 +0000 UTC m=+6450.766929117" watchObservedRunningTime="2026-11-02 19:11:03.693797804 +0000 UTC m=+6450.767325007" Nov 02 19:11:05.946683 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:11:05.946640 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:11:09.587860 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:09.587820 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:11:12.345916 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:12.345877 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:11:14.050196 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:14.050157 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw"] Nov 02 19:11:14.100652 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:14.100619 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-bqsmw"] Nov 02 19:11:15.080696 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:15.080661 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1e65fa-31ba-40ee-b273-730827a223ff" path="/var/lib/kubelet/pods/ad1e65fa-31ba-40ee-b273-730827a223ff/volumes" Nov 02 19:11:26.045474 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:26.045436 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc"] Nov 02 19:11:26.095497 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:26.095463 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-4x4tc"] Nov 02 19:11:27.080326 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:27.080291 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7" path="/var/lib/kubelet/pods/c8c24d7e-5c00-4f05-a3d8-ef2bfdd1d5d7/volumes" Nov 02 19:11:29.130674 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:29.130637 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:11:35.946840 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:11:35.946796 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:11:38.519367 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:38.519334 5647 scope.go:122] "RemoveContainer" containerID="08cf5005aeb06e482d8037b091a34c0ff3cb7ca41f6832fb809e52161cd982fe" Nov 02 19:11:48.779616 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:11:48.779527 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:12:01.044455 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:01.044407 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt"] Nov 02 19:12:01.094978 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:01.094944 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-b99dt"] Nov 02 19:12:03.079877 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:03.079839 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="283b2c0d-031c-45e6-bcf2-e5c99f1ae36e" path="/var/lib/kubelet/pods/283b2c0d-031c-45e6-bcf2-e5c99f1ae36e/volumes" Nov 02 19:12:05.946184 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:12:05.946145 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:12:12.050895 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:12.050858 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp"] Nov 02 19:12:12.100937 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:12.100901 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-trlwp"] Nov 02 19:12:13.080023 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:13.079991 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7791d99-2f83-4249-8952-3e660405c723" path="/var/lib/kubelet/pods/d7791d99-2f83-4249-8952-3e660405c723/volumes" Nov 02 19:12:18.159260 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:18.159206 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:12:26.241864 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:26.241825 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r"] Nov 02 19:12:26.242469 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:26.242030 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" podUID="244bbfd0-a739-42b8-a3bb-e225bc49a43e" containerName="maas-api" containerID="cri-o://e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38" gracePeriod=30 Nov 02 19:12:27.500575 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.500547 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:12:27.680845 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.680807 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk\" (UniqueName: \"kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk\") pod \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " Nov 02 19:12:27.681041 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.680885 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca\") pod \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " Nov 02 19:12:27.681041 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.680947 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs\") pod \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " Nov 02 19:12:27.681041 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.680980 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\") pod \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\" (UID: \"244bbfd0-a739-42b8-a3bb-e225bc49a43e\") " Nov 02 19:12:27.681337 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.681290 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca" pod "244bbfd0-a739-42b8-a3bb-e225bc49a43e" (UID: "244bbfd0-a739-42b8-a3bb-e225bc49a43e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:12:27.683150 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.683121 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk" pod "244bbfd0-a739-42b8-a3bb-e225bc49a43e" (UID: "244bbfd0-a739-42b8-a3bb-e225bc49a43e"). InnerVolumeSpecName "kube-api-access-gnslk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:12:27.683272 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.683193 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls" pod "244bbfd0-a739-42b8-a3bb-e225bc49a43e" (UID: "244bbfd0-a739-42b8-a3bb-e225bc49a43e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:12:27.683272 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.683208 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs" pod "244bbfd0-a739-42b8-a3bb-e225bc49a43e" (UID: "244bbfd0-a739-42b8-a3bb-e225bc49a43e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:12:27.781905 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.781868 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnslk\" (UniqueName: \"kubernetes.io/projected/244bbfd0-a739-42b8-a3bb-e225bc49a43e-kube-api-access-gnslk\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:12:27.781905 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.781899 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/244bbfd0-a739-42b8-a3bb-e225bc49a43e-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:12:27.781905 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.781909 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:12:27.782167 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:27.781919 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/244bbfd0-a739-42b8-a3bb-e225bc49a43e-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:12:28.039290 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.039203 5647 generic.go:356] "Generic (PLEG): container finished" podID="244bbfd0-a739-42b8-a3bb-e225bc49a43e" containerID="e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38" exitCode=0 Nov 02 19:12:28.039290 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.039247 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" event={"ID":"244bbfd0-a739-42b8-a3bb-e225bc49a43e","Type":"ContainerDied","Data":"e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38"} Nov 02 19:12:28.039290 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.039259 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" Nov 02 19:12:28.039537 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.039297 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r" event={"ID":"244bbfd0-a739-42b8-a3bb-e225bc49a43e","Type":"ContainerDied","Data":"25a0bc72f574e06476a527416a44b1c4b40c1b1c3999e21271667191cde8cd71"} Nov 02 19:12:28.039537 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.039319 5647 scope.go:122] "RemoveContainer" containerID="e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38" Nov 02 19:12:28.049951 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.049927 5647 scope.go:122] "RemoveContainer" containerID="e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38" Nov 02 19:12:28.050244 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:12:28.050222 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38\": container with ID starting with e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38 not found: ID does not exist" containerID="e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38" Nov 02 19:12:28.050324 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.050252 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38"} err="failed to get container status \"e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38\": rpc error: code = NotFound desc = could not find container \"e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38\": container with ID starting with e4e35d1b78f2ab3135a8e80783b06f45178c69190f1b4cb83bed084d9bb9bc38 not found: ID does not exist" Nov 02 19:12:28.204797 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.204761 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r"] Nov 02 19:12:28.256082 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:28.256048 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-5b22de-5d69b98665-rpd6r"] Nov 02 19:12:29.080528 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:29.080493 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244bbfd0-a739-42b8-a3bb-e225bc49a43e" path="/var/lib/kubelet/pods/244bbfd0-a739-42b8-a3bb-e225bc49a43e/volumes" Nov 02 19:12:30.044666 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:30.044631 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:12:35.946976 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:12:35.946929 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:12:38.543027 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:38.542991 5647 scope.go:122] "RemoveContainer" containerID="7deb412af042ed7a0d0e987dcc7d6e02e245d122b668d67470ae443334ec13c1" Nov 02 19:12:38.551825 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:38.551796 5647 scope.go:122] "RemoveContainer" containerID="c325517b5e18fc6d7dcfc129ac2288c5ec360292f84f24e149c56c646f7de135" Nov 02 19:12:38.561333 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:38.561306 5647 scope.go:122] "RemoveContainer" containerID="4c1e8e9eaee1d4d87ecc78ecba8f43c2cb663f6a85391c0904c9ce7418afa15e" Nov 02 19:12:47.043526 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:47.043492 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4"] Nov 02 19:12:47.093774 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:47.093743 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8kgt4"] Nov 02 19:12:49.080294 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:12:49.080258 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4aa2c0c-f914-412a-a975-d59159e137aa" path="/var/lib/kubelet/pods/d4aa2c0c-f914-412a-a975-d59159e137aa/volumes" Nov 02 19:13:05.946769 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:13:05.946710 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:13:07.767238 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:07.767206 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:13:09.045079 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:09.045035 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6"] Nov 02 19:13:09.095393 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:09.095361 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-wvzg6"] Nov 02 19:13:11.080227 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.080186 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3811f47-bf4d-4960-a4fa-de2ffba7ffec" path="/var/lib/kubelet/pods/a3811f47-bf4d-4960-a4fa-de2ffba7ffec/volumes" Nov 02 19:13:11.360608 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.360568 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r"] Nov 02 19:13:11.361091 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.361076 5647 cpu_manager.go:414] "containerMap: removing container" podUID="244bbfd0-a739-42b8-a3bb-e225bc49a43e" containerName="maas-api" Nov 02 19:13:11.361154 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.361093 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244bbfd0-a739-42b8-a3bb-e225bc49a43e" containerName="maas-api" Nov 02 19:13:11.361200 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.361180 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="244bbfd0-a739-42b8-a3bb-e225bc49a43e" containerName="maas-api" Nov 02 19:13:11.364440 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.364424 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.420701 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.420643 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:13:11.422934 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.422909 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r"] Nov 02 19:13:11.433074 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.433048 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.433215 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.433094 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhch\" (UniqueName: \"kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.433263 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.433243 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.471582 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.471547 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:13:11.534544 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.534508 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhch\" (UniqueName: \"kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.534783 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.534590 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.534783 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.534643 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.535247 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.535221 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.537189 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.537166 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.591986 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.591956 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhch\" (UniqueName: \"kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch\") pod \"maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:11.677134 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:11.677046 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:12.047757 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:12.047712 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r"] Nov 02 19:13:12.049849 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:13:12.049808 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8be401_0783_4562_824f_95858dc47d1f.slice/crio-b669f7a9924837d75fa33c6f6553865ac3c1f5ecea3234264eb7960216dc2525 WatchSource:0}: Error finding container b669f7a9924837d75fa33c6f6553865ac3c1f5ecea3234264eb7960216dc2525: Status 404 returned error can't find the container with id b669f7a9924837d75fa33c6f6553865ac3c1f5ecea3234264eb7960216dc2525 Nov 02 19:13:12.274084 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:12.274044 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" event={"ID":"7a8be401-0783-4562-824f-95858dc47d1f","Type":"ContainerStarted","Data":"b669f7a9924837d75fa33c6f6553865ac3c1f5ecea3234264eb7960216dc2525"} Nov 02 19:13:13.281245 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:13.281152 5647 generic.go:356] "Generic (PLEG): container finished" podID="7a8be401-0783-4562-824f-95858dc47d1f" containerID="838b5a2d94da9fa1d176081c8eefa0cf05b66a8684927cb9b32656299f0175e2" exitCode=0 Nov 02 19:13:13.281245 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:13.281224 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" event={"ID":"7a8be401-0783-4562-824f-95858dc47d1f","Type":"ContainerDied","Data":"838b5a2d94da9fa1d176081c8eefa0cf05b66a8684927cb9b32656299f0175e2"} Nov 02 19:13:14.423038 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.423012 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:14.465244 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.465207 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca\") pod \"7a8be401-0783-4562-824f-95858dc47d1f\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " Nov 02 19:13:14.465442 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.465325 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca\") pod \"7a8be401-0783-4562-824f-95858dc47d1f\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " Nov 02 19:13:14.465442 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.465364 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch\" (UniqueName: \"kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch\") pod \"7a8be401-0783-4562-824f-95858dc47d1f\" (UID: \"7a8be401-0783-4562-824f-95858dc47d1f\") " Nov 02 19:13:14.465777 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.465720 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca" pod "7a8be401-0783-4562-824f-95858dc47d1f" (UID: "7a8be401-0783-4562-824f-95858dc47d1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:13:14.467529 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.467491 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca" pod "7a8be401-0783-4562-824f-95858dc47d1f" (UID: "7a8be401-0783-4562-824f-95858dc47d1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:13:14.467624 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.467608 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch" pod "7a8be401-0783-4562-824f-95858dc47d1f" (UID: "7a8be401-0783-4562-824f-95858dc47d1f"). InnerVolumeSpecName "kube-api-access-vbhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:13:14.566762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.566651 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:13:14.566762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.566691 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a8be401-0783-4562-824f-95858dc47d1f-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:13:14.566762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:14.566707 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhch\" (UniqueName: \"kubernetes.io/projected/7a8be401-0783-4562-824f-95858dc47d1f-kube-api-access-vbhch\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:13:15.294075 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:15.294039 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" event={"ID":"7a8be401-0783-4562-824f-95858dc47d1f","Type":"ContainerDied","Data":"b669f7a9924837d75fa33c6f6553865ac3c1f5ecea3234264eb7960216dc2525"} Nov 02 19:13:15.294075 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:15.294063 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r" Nov 02 19:13:15.294305 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:15.294094 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b669f7a9924837d75fa33c6f6553865ac3c1f5ecea3234264eb7960216dc2525" Nov 02 19:13:30.060514 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:30.060473 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw"] Nov 02 19:13:30.096798 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:30.096760 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7s5dw"] Nov 02 19:13:30.326325 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:30.326226 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r"] Nov 02 19:13:30.329285 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:30.329261 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-5b22de-snd2r"] Nov 02 19:13:31.080422 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:31.080379 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c27f38-2846-456a-b13e-2d2d81bcde32" path="/var/lib/kubelet/pods/38c27f38-2846-456a-b13e-2d2d81bcde32/volumes" Nov 02 19:13:31.081036 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:31.081014 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a8be401-0783-4562-824f-95858dc47d1f" path="/var/lib/kubelet/pods/7a8be401-0783-4562-824f-95858dc47d1f/volumes" Nov 02 19:13:32.349771 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.349722 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb"] Nov 02 19:13:32.400447 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.400414 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fmdkb"] Nov 02 19:13:32.470057 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.470019 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p"] Nov 02 19:13:32.470528 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.470510 5647 cpu_manager.go:414] "containerMap: removing container" podUID="7a8be401-0783-4562-824f-95858dc47d1f" containerName="revoke-keys" Nov 02 19:13:32.470639 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.470529 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a8be401-0783-4562-824f-95858dc47d1f" containerName="revoke-keys" Nov 02 19:13:32.470704 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.470649 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a8be401-0783-4562-824f-95858dc47d1f" containerName="revoke-keys" Nov 02 19:13:32.473835 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.473814 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.528073 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.528047 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:13:32.528715 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.528689 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:13:32.530265 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.530244 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p"] Nov 02 19:13:32.552352 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.552320 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.552535 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.552393 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smrts\" (UniqueName: \"kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.552535 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.552484 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.654222 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.654136 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.654222 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.654184 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smrts\" (UniqueName: \"kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.654478 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.654452 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.654877 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.654858 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.656932 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.656909 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.711993 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.711954 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smrts\" (UniqueName: \"kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts\") pod \"maas-api-revoke-sub-3d4156f1-t7j6p\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:32.785861 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:32.785818 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:33.081269 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:33.081177 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35e905ab-1bcd-46ea-b24c-5b606dcb9053" path="/var/lib/kubelet/pods/35e905ab-1bcd-46ea-b24c-5b606dcb9053/volumes" Nov 02 19:13:33.162849 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:33.162826 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p"] Nov 02 19:13:33.165207 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:13:33.165178 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e486ed1_cc19_415b_91ed_e821e2d88bf0.slice/crio-bf0feb4faeb0c9ea3b4e15b7899637630192f5da29c605fb3d7ae9c66c9d51f5 WatchSource:0}: Error finding container bf0feb4faeb0c9ea3b4e15b7899637630192f5da29c605fb3d7ae9c66c9d51f5: Status 404 returned error can't find the container with id bf0feb4faeb0c9ea3b4e15b7899637630192f5da29c605fb3d7ae9c66c9d51f5 Nov 02 19:13:33.393063 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:33.393030 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" event={"ID":"3e486ed1-cc19-415b-91ed-e821e2d88bf0","Type":"ContainerStarted","Data":"bf0feb4faeb0c9ea3b4e15b7899637630192f5da29c605fb3d7ae9c66c9d51f5"} Nov 02 19:13:34.333113 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:34.333023 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:13:34.400575 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:34.400543 5647 generic.go:356] "Generic (PLEG): container finished" podID="3e486ed1-cc19-415b-91ed-e821e2d88bf0" containerID="b38eb8adada7222422004d653ffc34eac75c26d5e96eb3b079b09c32da1b723c" exitCode=0 Nov 02 19:13:34.401057 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:34.400637 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" event={"ID":"3e486ed1-cc19-415b-91ed-e821e2d88bf0","Type":"ContainerDied","Data":"b38eb8adada7222422004d653ffc34eac75c26d5e96eb3b079b09c32da1b723c"} Nov 02 19:13:35.121051 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.121026 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:13:35.131225 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.131198 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:13:35.546911 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.546886 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:35.685323 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.685224 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca\") pod \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " Nov 02 19:13:35.685323 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.685298 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts\" (UniqueName: \"kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts\") pod \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " Nov 02 19:13:35.685584 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.685366 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca\") pod \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\" (UID: \"3e486ed1-cc19-415b-91ed-e821e2d88bf0\") " Nov 02 19:13:35.685722 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.685697 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca" pod "3e486ed1-cc19-415b-91ed-e821e2d88bf0" (UID: "3e486ed1-cc19-415b-91ed-e821e2d88bf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:13:35.687680 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.687649 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts" pod "3e486ed1-cc19-415b-91ed-e821e2d88bf0" (UID: "3e486ed1-cc19-415b-91ed-e821e2d88bf0"). InnerVolumeSpecName "kube-api-access-smrts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:13:35.687680 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.687660 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca" pod "3e486ed1-cc19-415b-91ed-e821e2d88bf0" (UID: "3e486ed1-cc19-415b-91ed-e821e2d88bf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:13:35.786620 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.786588 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:13:35.786620 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.786614 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smrts\" (UniqueName: \"kubernetes.io/projected/3e486ed1-cc19-415b-91ed-e821e2d88bf0-kube-api-access-smrts\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:13:35.786620 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:35.786625 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e486ed1-cc19-415b-91ed-e821e2d88bf0-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:13:35.947439 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:13:35.947345 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:13:36.412217 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:36.412182 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" Nov 02 19:13:36.412217 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:36.412208 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p" event={"ID":"3e486ed1-cc19-415b-91ed-e821e2d88bf0","Type":"ContainerDied","Data":"bf0feb4faeb0c9ea3b4e15b7899637630192f5da29c605fb3d7ae9c66c9d51f5"} Nov 02 19:13:36.412486 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:36.412255 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0feb4faeb0c9ea3b4e15b7899637630192f5da29c605fb3d7ae9c66c9d51f5" Nov 02 19:13:38.596693 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:38.596662 5647 scope.go:122] "RemoveContainer" containerID="30f6988771e7bdb47584193c443f35f72789b19b6b1c04f6d374e637ecadd2dc" Nov 02 19:13:38.605913 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:38.605890 5647 scope.go:122] "RemoveContainer" containerID="79f7ad411d36de3a86260d0a228079f3a7a7e43c8abd1a9394a0873a71d145f2" Nov 02 19:13:46.859743 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:46.859693 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:13:48.055970 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:48.055924 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph"] Nov 02 19:13:48.101762 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:48.100774 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-85qph"] Nov 02 19:13:48.499918 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:48.499876 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:13:48.500232 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:48.500196 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:13:49.100759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:13:49.100261 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77630e73-3f8d-4bed-bef7-740154add03f" path="/var/lib/kubelet/pods/77630e73-3f8d-4bed-bef7-740154add03f/volumes" Nov 02 19:14:05.946927 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:14:05.946889 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:14:35.947915 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:14:35.947871 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:14:38.632358 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:14:38.632324 5647 scope.go:122] "RemoveContainer" containerID="4fc91e61bb21889d39eccb004cda5722db96ec54549a15a4e236c89d7a09aa30" Nov 02 19:14:38.641620 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:14:38.641586 5647 scope.go:122] "RemoveContainer" containerID="2c53f87253f783f9d0f58d8553963aa5aa3149d2966eb815888d6d5aa3412960" Nov 02 19:14:38.652186 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:14:38.652165 5647 scope.go:122] "RemoveContainer" containerID="8d20a48196d2e8d785eba84c3fa5c9f7efc0621ca2dbf5cd82054ea80b7060c0" Nov 02 19:15:00.153778 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.153686 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs"] Nov 02 19:15:00.154312 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.154176 5647 cpu_manager.go:414] "containerMap: removing container" podUID="3e486ed1-cc19-415b-91ed-e821e2d88bf0" containerName="revoke-keys" Nov 02 19:15:00.154312 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.154191 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e486ed1-cc19-415b-91ed-e821e2d88bf0" containerName="revoke-keys" Nov 02 19:15:00.154312 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.154276 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e486ed1-cc19-415b-91ed-e821e2d88bf0" containerName="revoke-keys" Nov 02 19:15:00.157496 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.157477 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.211417 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.211378 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:15:00.213300 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.213270 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs"] Nov 02 19:15:00.270269 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.270134 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894115-g8jjs\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.270524 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.270386 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz2tx\" (UniqueName: \"kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx\") pod \"maas-api-key-cleanup-29894115-g8jjs\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.371527 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.371489 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894115-g8jjs\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.371719 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.371562 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz2tx\" (UniqueName: \"kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx\") pod \"maas-api-key-cleanup-29894115-g8jjs\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.372181 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.372161 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894115-g8jjs\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.429673 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.429602 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz2tx\" (UniqueName: \"kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx\") pod \"maas-api-key-cleanup-29894115-g8jjs\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.471172 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.471137 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:00.842899 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.842874 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs"] Nov 02 19:15:00.845029 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:15:00.845004 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd87d0b_7f7a_4b5b_a581_89b0255e90f3.slice/crio-804a6dc10d8892a889ee5669c3f41e1bccadd879d27b5cd6d0c7a08cdd7092b5 WatchSource:0}: Error finding container 804a6dc10d8892a889ee5669c3f41e1bccadd879d27b5cd6d0c7a08cdd7092b5: Status 404 returned error can't find the container with id 804a6dc10d8892a889ee5669c3f41e1bccadd879d27b5cd6d0c7a08cdd7092b5 Nov 02 19:15:00.921680 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:00.921647 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" event={"ID":"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3","Type":"ContainerStarted","Data":"804a6dc10d8892a889ee5669c3f41e1bccadd879d27b5cd6d0c7a08cdd7092b5"} Nov 02 19:15:01.928390 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:01.928298 5647 generic.go:356] "Generic (PLEG): container finished" podID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerID="1fa4466e0d19d9455b56760d1e35117d769c0824b331cc31819e4a53e84e2b5f" exitCode=60 Nov 02 19:15:01.928390 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:01.928353 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" event={"ID":"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3","Type":"ContainerDied","Data":"1fa4466e0d19d9455b56760d1e35117d769c0824b331cc31819e4a53e84e2b5f"} Nov 02 19:15:01.928856 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:01.928611 5647 scope.go:122] "RemoveContainer" containerID="1fa4466e0d19d9455b56760d1e35117d769c0824b331cc31819e4a53e84e2b5f" Nov 02 19:15:02.935651 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:02.935611 5647 generic.go:356] "Generic (PLEG): container finished" podID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerID="955d9e63fba40a9d761531596ecd5f5c27739c89cde599581285e18eb43e377f" exitCode=60 Nov 02 19:15:02.936151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:02.935660 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" event={"ID":"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3","Type":"ContainerDied","Data":"955d9e63fba40a9d761531596ecd5f5c27739c89cde599581285e18eb43e377f"} Nov 02 19:15:02.936151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:02.935715 5647 scope.go:122] "RemoveContainer" containerID="1fa4466e0d19d9455b56760d1e35117d769c0824b331cc31819e4a53e84e2b5f" Nov 02 19:15:02.936151 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:02.935977 5647 scope.go:122] "RemoveContainer" containerID="955d9e63fba40a9d761531596ecd5f5c27739c89cde599581285e18eb43e377f" Nov 02 19:15:02.936287 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:15:02.936244 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894115-g8jjs_odh-ai-gateway-infra(2cd87d0b-7f7a-4b5b-a581-89b0255e90f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" Nov 02 19:15:03.943136 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:03.943106 5647 scope.go:122] "RemoveContainer" containerID="955d9e63fba40a9d761531596ecd5f5c27739c89cde599581285e18eb43e377f" Nov 02 19:15:03.943539 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:15:03.943360 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894115-g8jjs_odh-ai-gateway-infra(2cd87d0b-7f7a-4b5b-a581-89b0255e90f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" Nov 02 19:15:05.947211 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:15:05.947169 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:15:08.056460 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:08.056425 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd"] Nov 02 19:15:08.107065 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:08.107029 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-9m9qd"] Nov 02 19:15:09.079641 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:09.079606 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0225158d-2be3-4485-9097-ab53db834898" path="/var/lib/kubelet/pods/0225158d-2be3-4485-9097-ab53db834898/volumes" Nov 02 19:15:15.074584 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:15.074543 5647 scope.go:122] "RemoveContainer" containerID="955d9e63fba40a9d761531596ecd5f5c27739c89cde599581285e18eb43e377f" Nov 02 19:15:16.007101 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:16.007067 5647 generic.go:356] "Generic (PLEG): container finished" podID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerID="8b158acfeba4a88493f8e15f48d9f47227cbcdad5a9949e94b4c3a77fc9361d9" exitCode=60 Nov 02 19:15:16.007278 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:16.007135 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" event={"ID":"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3","Type":"ContainerDied","Data":"8b158acfeba4a88493f8e15f48d9f47227cbcdad5a9949e94b4c3a77fc9361d9"} Nov 02 19:15:16.007278 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:16.007196 5647 scope.go:122] "RemoveContainer" containerID="955d9e63fba40a9d761531596ecd5f5c27739c89cde599581285e18eb43e377f" Nov 02 19:15:16.007605 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:16.007588 5647 scope.go:122] "RemoveContainer" containerID="8b158acfeba4a88493f8e15f48d9f47227cbcdad5a9949e94b4c3a77fc9361d9" Nov 02 19:15:16.007910 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:15:16.007891 5647 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894115-g8jjs_odh-ai-gateway-infra(2cd87d0b-7f7a-4b5b-a581-89b0255e90f3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" Nov 02 19:15:17.135498 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.135460 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs"] Nov 02 19:15:17.371325 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.371290 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:17.446571 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.446458 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx\" (UniqueName: \"kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx\") pod \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " Nov 02 19:15:17.446571 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.446541 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca\") pod \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\" (UID: \"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3\") " Nov 02 19:15:17.447143 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.447112 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca" pod "2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" (UID: "2cd87d0b-7f7a-4b5b-a581-89b0255e90f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:15:17.447937 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.447915 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-maas-api-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:17.449452 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.449421 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx" pod "2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" (UID: "2cd87d0b-7f7a-4b5b-a581-89b0255e90f3"). InnerVolumeSpecName "kube-api-access-qz2tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:15:17.549202 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:17.549140 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz2tx\" (UniqueName: \"kubernetes.io/projected/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3-kube-api-access-qz2tx\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:18.081781 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:18.081753 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" Nov 02 19:15:18.131572 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:18.131534 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs" event={"ID":"2cd87d0b-7f7a-4b5b-a581-89b0255e90f3","Type":"ContainerDied","Data":"804a6dc10d8892a889ee5669c3f41e1bccadd879d27b5cd6d0c7a08cdd7092b5"} Nov 02 19:15:18.131873 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:18.131846 5647 scope.go:122] "RemoveContainer" containerID="8b158acfeba4a88493f8e15f48d9f47227cbcdad5a9949e94b4c3a77fc9361d9" Nov 02 19:15:18.266760 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:18.265769 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs"] Nov 02 19:15:18.274555 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:18.274501 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894115-g8jjs"] Nov 02 19:15:19.052761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:19.051788 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw"] Nov 02 19:15:19.052999 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:19.052908 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-d6kcw"] Nov 02 19:15:19.098927 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:19.098885 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" path="/var/lib/kubelet/pods/2cd87d0b-7f7a-4b5b-a581-89b0255e90f3/volumes" Nov 02 19:15:19.099903 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:19.099876 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4730a3-5299-4f38-98c6-544bf7c7d47f" path="/var/lib/kubelet/pods/ba4730a3-5299-4f38-98c6-544bf7c7d47f/volumes" Nov 02 19:15:21.430316 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:21.430283 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:15:23.446729 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.446695 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw"] Nov 02 19:15:23.447250 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447133 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.447250 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447147 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.447250 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447170 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.447250 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447175 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.447522 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447274 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.447522 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447283 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.447522 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.447293 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:23.454238 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.454216 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.507168 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.507134 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:15:23.507168 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.507137 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw"] Nov 02 19:15:23.555710 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.555677 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:15:23.565922 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.565888 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgc8\" (UniqueName: \"kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.565922 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.565922 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.566164 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.565958 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.667497 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.667461 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgc8\" (UniqueName: \"kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.667675 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.667501 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.667675 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.667546 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.668232 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.668209 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.670061 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.670041 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.725497 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.725423 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgc8\" (UniqueName: \"kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8\") pod \"maas-api-revoke-sub-d576e68b-glclw\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:23.767059 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:23.767022 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:24.136991 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:24.136965 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw"] Nov 02 19:15:24.139652 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:15:24.139618 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc66f43_024d_4b5b_aae9_6f79278d5661.slice/crio-b4d337e4410144c091eb6bb000f37ee02fbd1435df6a0b868dad3907ce5b06dd WatchSource:0}: Error finding container b4d337e4410144c091eb6bb000f37ee02fbd1435df6a0b868dad3907ce5b06dd: Status 404 returned error can't find the container with id b4d337e4410144c091eb6bb000f37ee02fbd1435df6a0b868dad3907ce5b06dd Nov 02 19:15:24.194805 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:24.194773 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" event={"ID":"6cc66f43-024d-4b5b-aae9-6f79278d5661","Type":"ContainerStarted","Data":"b4d337e4410144c091eb6bb000f37ee02fbd1435df6a0b868dad3907ce5b06dd"} Nov 02 19:15:25.202571 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:25.202536 5647 generic.go:356] "Generic (PLEG): container finished" podID="6cc66f43-024d-4b5b-aae9-6f79278d5661" containerID="34c6cd88743247b9ae82919ec0de6e2f3b6528564f2fad8c0236d404b8fb985c" exitCode=0 Nov 02 19:15:25.202571 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:25.202572 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" event={"ID":"6cc66f43-024d-4b5b-aae9-6f79278d5661","Type":"ContainerDied","Data":"34c6cd88743247b9ae82919ec0de6e2f3b6528564f2fad8c0236d404b8fb985c"} Nov 02 19:15:26.338710 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.338685 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:26.499852 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.499752 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca\") pod \"6cc66f43-024d-4b5b-aae9-6f79278d5661\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " Nov 02 19:15:26.499852 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.499796 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca\") pod \"6cc66f43-024d-4b5b-aae9-6f79278d5661\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " Nov 02 19:15:26.499852 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.499841 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8\" (UniqueName: \"kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8\") pod \"6cc66f43-024d-4b5b-aae9-6f79278d5661\" (UID: \"6cc66f43-024d-4b5b-aae9-6f79278d5661\") " Nov 02 19:15:26.500185 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.500138 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca" pod "6cc66f43-024d-4b5b-aae9-6f79278d5661" (UID: "6cc66f43-024d-4b5b-aae9-6f79278d5661"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:15:26.502075 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.502040 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca" pod "6cc66f43-024d-4b5b-aae9-6f79278d5661" (UID: "6cc66f43-024d-4b5b-aae9-6f79278d5661"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:15:26.502199 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.502166 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8" pod "6cc66f43-024d-4b5b-aae9-6f79278d5661" (UID: "6cc66f43-024d-4b5b-aae9-6f79278d5661"). InnerVolumeSpecName "kube-api-access-7wgc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:15:26.601095 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.601062 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:26.601095 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.601093 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc66f43-024d-4b5b-aae9-6f79278d5661-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:26.601095 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:26.601104 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgc8\" (UniqueName: \"kubernetes.io/projected/6cc66f43-024d-4b5b-aae9-6f79278d5661-kube-api-access-7wgc8\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:27.214724 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:27.214697 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" Nov 02 19:15:27.214917 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:27.214748 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw" event={"ID":"6cc66f43-024d-4b5b-aae9-6f79278d5661","Type":"ContainerDied","Data":"b4d337e4410144c091eb6bb000f37ee02fbd1435df6a0b868dad3907ce5b06dd"} Nov 02 19:15:27.214917 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:27.214797 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4d337e4410144c091eb6bb000f37ee02fbd1435df6a0b868dad3907ce5b06dd" Nov 02 19:15:35.947293 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:15:35.947251 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:15:36.166074 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:36.166037 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:15:38.685763 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:38.685694 5647 scope.go:122] "RemoveContainer" containerID="e42bbadd637eff07ff7ee02658935b381b80310f5ea001989537528cd0e716cd" Nov 02 19:15:38.696924 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:38.696899 5647 scope.go:122] "RemoveContainer" containerID="8b560aa42472cf566f80fb7aefd279dd1bf5abb7e1c5873f9708b577bc4b2ec3" Nov 02 19:15:38.705980 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:38.705956 5647 scope.go:122] "RemoveContainer" containerID="e99f84f1ad530f13b5f59a73a3832dbd88a48a6d41099a12f6c27d4ff2f69595" Nov 02 19:15:45.183792 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.183757 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs"] Nov 02 19:15:45.184220 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.184177 5647 cpu_manager.go:414] "containerMap: removing container" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:45.184220 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.184188 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd87d0b-7f7a-4b5b-a581-89b0255e90f3" containerName="cleanup" Nov 02 19:15:45.184220 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.184202 5647 cpu_manager.go:414] "containerMap: removing container" podUID="6cc66f43-024d-4b5b-aae9-6f79278d5661" containerName="revoke-keys" Nov 02 19:15:45.184220 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.184207 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc66f43-024d-4b5b-aae9-6f79278d5661" containerName="revoke-keys" Nov 02 19:15:45.184366 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.184302 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc66f43-024d-4b5b-aae9-6f79278d5661" containerName="revoke-keys" Nov 02 19:15:45.188976 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.188951 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.243250 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.243218 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs"] Nov 02 19:15:45.275102 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.275068 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.275102 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.275106 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcdm2\" (UniqueName: \"kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.275403 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.275206 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.291892 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.291849 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 19:15:45.291892 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.291873 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nq2mt\"" Nov 02 19:15:45.376087 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.376049 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.376296 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.376094 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcdm2\" (UniqueName: \"kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.376296 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.376158 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.376784 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.376758 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.378623 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.378605 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.433688 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.433654 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcdm2\" (UniqueName: \"kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2\") pod \"maas-api-revoke-sub-d2bc93bc-5d5cs\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.503169 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.503060 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:45.874527 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.874501 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs"] Nov 02 19:15:45.876589 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:15:45.876563 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44bca6b_b1e5_47ce_9f8d_cd72e440117c.slice/crio-0b0104f21634576fac3e24ef634670aa326325485670b19d22a78321d7a08bc8 WatchSource:0}: Error finding container 0b0104f21634576fac3e24ef634670aa326325485670b19d22a78321d7a08bc8: Status 404 returned error can't find the container with id 0b0104f21634576fac3e24ef634670aa326325485670b19d22a78321d7a08bc8 Nov 02 19:15:45.972766 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:45.972719 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:15:46.376331 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:46.376292 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" event={"ID":"b44bca6b-b1e5-47ce-9f8d-cd72e440117c","Type":"ContainerStarted","Data":"06a9526faa3e7705e29b7232da96fc9b3ba923d234116ff958cb5d9faa56ab54"} Nov 02 19:15:46.376719 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:46.376352 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" event={"ID":"b44bca6b-b1e5-47ce-9f8d-cd72e440117c","Type":"ContainerStarted","Data":"0b0104f21634576fac3e24ef634670aa326325485670b19d22a78321d7a08bc8"} Nov 02 19:15:46.490225 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:46.490129 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" podStartSLOduration=1.490116436 podStartE2EDuration="1.490116436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:15:46.490017372 +0000 UTC m=+6733.563544575" watchObservedRunningTime="2026-11-02 19:15:46.490116436 +0000 UTC m=+6733.563643638" Nov 02 19:15:47.383842 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:47.383807 5647 generic.go:356] "Generic (PLEG): container finished" podID="b44bca6b-b1e5-47ce-9f8d-cd72e440117c" containerID="06a9526faa3e7705e29b7232da96fc9b3ba923d234116ff958cb5d9faa56ab54" exitCode=0 Nov 02 19:15:47.384377 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:47.383846 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" event={"ID":"b44bca6b-b1e5-47ce-9f8d-cd72e440117c","Type":"ContainerDied","Data":"06a9526faa3e7705e29b7232da96fc9b3ba923d234116ff958cb5d9faa56ab54"} Nov 02 19:15:48.529501 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.529473 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:48.608363 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.608332 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca\") pod \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " Nov 02 19:15:48.608587 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.608426 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca\") pod \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " Nov 02 19:15:48.608587 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.608452 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2\" (UniqueName: \"kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2\") pod \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\" (UID: \"b44bca6b-b1e5-47ce-9f8d-cd72e440117c\") " Nov 02 19:15:48.608885 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.608860 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca" pod "b44bca6b-b1e5-47ce-9f8d-cd72e440117c" (UID: "b44bca6b-b1e5-47ce-9f8d-cd72e440117c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:15:48.610723 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.610698 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2" pod "b44bca6b-b1e5-47ce-9f8d-cd72e440117c" (UID: "b44bca6b-b1e5-47ce-9f8d-cd72e440117c"). InnerVolumeSpecName "kube-api-access-lcdm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:15:48.610818 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.610796 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca" pod "b44bca6b-b1e5-47ce-9f8d-cd72e440117c" (UID: "b44bca6b-b1e5-47ce-9f8d-cd72e440117c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:15:48.710397 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.710315 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-xks-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:48.710397 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.710345 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-maas-api-openshift-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:48.710397 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:48.710354 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcdm2\" (UniqueName: \"kubernetes.io/projected/b44bca6b-b1e5-47ce-9f8d-cd72e440117c-kube-api-access-lcdm2\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:15:49.395911 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:49.395823 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" Nov 02 19:15:49.395911 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:49.395844 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs" event={"ID":"b44bca6b-b1e5-47ce-9f8d-cd72e440117c","Type":"ContainerDied","Data":"0b0104f21634576fac3e24ef634670aa326325485670b19d22a78321d7a08bc8"} Nov 02 19:15:49.395911 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:49.395907 5647 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b0104f21634576fac3e24ef634670aa326325485670b19d22a78321d7a08bc8" Nov 02 19:15:54.278989 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:54.278952 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:15:58.157376 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.157336 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx"] Nov 02 19:15:58.157846 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.157830 5647 cpu_manager.go:414] "containerMap: removing container" podUID="b44bca6b-b1e5-47ce-9f8d-cd72e440117c" containerName="revoke-keys" Nov 02 19:15:58.157901 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.157847 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44bca6b-b1e5-47ce-9f8d-cd72e440117c" containerName="revoke-keys" Nov 02 19:15:58.157948 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.157912 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44bca6b-b1e5-47ce-9f8d-cd72e440117c" containerName="revoke-keys" Nov 02 19:15:58.161003 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.160988 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.210624 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.210591 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.210801 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.210668 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9lfv\" (UniqueName: \"kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.211044 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.210988 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.211201 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.211081 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.266937 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.266826 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 19:15:58.266937 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.266821 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 19:15:58.266937 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.266890 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-94n2w\"" Nov 02 19:15:58.269505 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.269476 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx"] Nov 02 19:15:58.312001 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.311963 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.312187 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.312018 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.312187 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.312062 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.312187 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.312097 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9lfv\" (UniqueName: \"kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.312905 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.312847 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.314652 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.314628 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.314780 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.314661 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.374827 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.374790 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9lfv\" (UniqueName: \"kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv\") pod \"maas-api-5cb67577d6-7mnbx\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.474572 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.474480 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:15:58.849925 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:58.849891 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx"] Nov 02 19:15:58.852837 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:15:58.852390 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe154b44_f772_40b2_b398_630f2c47640d.slice/crio-60708c54d62666c78855a1ecf80948f58c107b91825e1b4c67a201b3b18ff081 WatchSource:0}: Error finding container 60708c54d62666c78855a1ecf80948f58c107b91825e1b4c67a201b3b18ff081: Status 404 returned error can't find the container with id 60708c54d62666c78855a1ecf80948f58c107b91825e1b4c67a201b3b18ff081 Nov 02 19:15:59.457785 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:15:59.457747 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" event={"ID":"be154b44-f772-40b2-b398-630f2c47640d","Type":"ContainerStarted","Data":"60708c54d62666c78855a1ecf80948f58c107b91825e1b4c67a201b3b18ff081"} Nov 02 19:16:00.464230 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:00.464191 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" event={"ID":"be154b44-f772-40b2-b398-630f2c47640d","Type":"ContainerStarted","Data":"19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e"} Nov 02 19:16:00.464646 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:00.464257 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:16:00.580560 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:00.580508 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" podStartSLOduration=1.5872248450000002 podStartE2EDuration="2.580492396s" totalImagesPullingTime="993.267551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:16:00.579170259 +0000 UTC m=+6747.652697475" watchObservedRunningTime="2026-11-02 19:16:00.580492396 +0000 UTC m=+6747.654019598" Nov 02 19:16:05.947639 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:16:05.947600 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:16:06.475215 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:06.475179 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:16:06.858338 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:06.858302 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv"] Nov 02 19:16:06.862221 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:06.862200 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:06.918552 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:06.918525 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv"] Nov 02 19:16:07.005578 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.005530 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123a44a0-c06c-4846-bc41-2359c7cfee38-maas-api-tls\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.005578 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.005586 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123a44a0-c06c-4846-bc41-2359c7cfee38-metrics-certs\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.006092 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.005697 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s556\" (UniqueName: \"kubernetes.io/projected/123a44a0-c06c-4846-bc41-2359c7cfee38-kube-api-access-9s556\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.006092 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.005755 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123a44a0-c06c-4846-bc41-2359c7cfee38-openshift-service-ca\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.106712 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.106658 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123a44a0-c06c-4846-bc41-2359c7cfee38-maas-api-tls\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.106712 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.106715 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123a44a0-c06c-4846-bc41-2359c7cfee38-metrics-certs\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.107049 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.106775 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s556\" (UniqueName: \"kubernetes.io/projected/123a44a0-c06c-4846-bc41-2359c7cfee38-kube-api-access-9s556\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.107049 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.106815 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123a44a0-c06c-4846-bc41-2359c7cfee38-openshift-service-ca\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.107430 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.107412 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123a44a0-c06c-4846-bc41-2359c7cfee38-openshift-service-ca\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.109362 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.109295 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123a44a0-c06c-4846-bc41-2359c7cfee38-maas-api-tls\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.109362 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.109332 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123a44a0-c06c-4846-bc41-2359c7cfee38-metrics-certs\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.164380 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.164340 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s556\" (UniqueName: \"kubernetes.io/projected/123a44a0-c06c-4846-bc41-2359c7cfee38-kube-api-access-9s556\") pod \"maas-api-5c87d67b66-lkmpv\" (UID: \"123a44a0-c06c-4846-bc41-2359c7cfee38\") " pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.175771 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.175716 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:07.552853 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.552819 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv"] Nov 02 19:16:07.554158 ip-10-0-128-71 kubenswrapper[5647]: W1102 19:16:07.554125 5647 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123a44a0_c06c_4846_bc41_2359c7cfee38.slice/crio-9ceeaa317c9df3bb4a20ed8e759686b343073e13c8aac1751a0060bf2f108b59 WatchSource:0}: Error finding container 9ceeaa317c9df3bb4a20ed8e759686b343073e13c8aac1751a0060bf2f108b59: Status 404 returned error can't find the container with id 9ceeaa317c9df3bb4a20ed8e759686b343073e13c8aac1751a0060bf2f108b59 Nov 02 19:16:07.555634 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:07.555615 5647 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 19:16:08.506154 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:08.506113 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" event={"ID":"123a44a0-c06c-4846-bc41-2359c7cfee38","Type":"ContainerStarted","Data":"2ad3d60ee96448eb5e16f37d87244bae7aedb82601e4be79d4518ebda8b2c7ae"} Nov 02 19:16:08.506154 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:08.506162 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" event={"ID":"123a44a0-c06c-4846-bc41-2359c7cfee38","Type":"ContainerStarted","Data":"9ceeaa317c9df3bb4a20ed8e759686b343073e13c8aac1751a0060bf2f108b59"} Nov 02 19:16:08.506647 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:08.506246 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:08.665277 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:08.665148 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" podStartSLOduration=1.8351804189999998 podStartE2EDuration="2.665132622s" totalImagesPullingTime="829.952203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 19:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:16:08.664419723 +0000 UTC m=+6755.737946926" watchObservedRunningTime="2026-11-02 19:16:08.665132622 +0000 UTC m=+6755.738659824" Nov 02 19:16:10.742549 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:10.742514 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:16:14.517161 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:14.517084 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c87d67b66-lkmpv" Nov 02 19:16:14.685093 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:14.685045 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx"] Nov 02 19:16:14.685329 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:14.685306 5647 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" podUID="be154b44-f772-40b2-b398-630f2c47640d" containerName="maas-api" containerID="cri-o://19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e" gracePeriod=30 Nov 02 19:16:14.945099 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:14.945074 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:16:15.048603 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.048560 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca\") pod \"be154b44-f772-40b2-b398-630f2c47640d\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " Nov 02 19:16:15.048816 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.048615 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv\" (UniqueName: \"kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv\") pod \"be154b44-f772-40b2-b398-630f2c47640d\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " Nov 02 19:16:15.048816 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.048663 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls\") pod \"be154b44-f772-40b2-b398-630f2c47640d\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " Nov 02 19:16:15.048816 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.048785 5647 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs\") pod \"be154b44-f772-40b2-b398-630f2c47640d\" (UID: \"be154b44-f772-40b2-b398-630f2c47640d\") " Nov 02 19:16:15.049048 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.049021 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca" pod "be154b44-f772-40b2-b398-630f2c47640d" (UID: "be154b44-f772-40b2-b398-630f2c47640d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 19:16:15.049208 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.049190 5647 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be154b44-f772-40b2-b398-630f2c47640d-openshift-service-ca\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:16:15.051002 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.050943 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv" pod "be154b44-f772-40b2-b398-630f2c47640d" (UID: "be154b44-f772-40b2-b398-630f2c47640d"). InnerVolumeSpecName "kube-api-access-z9lfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 19:16:15.051002 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.050951 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs" pod "be154b44-f772-40b2-b398-630f2c47640d" (UID: "be154b44-f772-40b2-b398-630f2c47640d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:16:15.051176 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.051115 5647 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls" pod "be154b44-f772-40b2-b398-630f2c47640d" (UID: "be154b44-f772-40b2-b398-630f2c47640d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 19:16:15.150277 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.150227 5647 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9lfv\" (UniqueName: \"kubernetes.io/projected/be154b44-f772-40b2-b398-630f2c47640d-kube-api-access-z9lfv\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:16:15.150277 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.150271 5647 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-maas-api-tls\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:16:15.150277 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.150285 5647 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be154b44-f772-40b2-b398-630f2c47640d-metrics-certs\") on node \"ip-10-0-128-71.us-west-1.compute.internal\" DevicePath \"\"" Nov 02 19:16:15.574174 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.574134 5647 generic.go:356] "Generic (PLEG): container finished" podID="be154b44-f772-40b2-b398-630f2c47640d" containerID="19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e" exitCode=0 Nov 02 19:16:15.574633 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.574182 5647 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" Nov 02 19:16:15.574633 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.574219 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" event={"ID":"be154b44-f772-40b2-b398-630f2c47640d","Type":"ContainerDied","Data":"19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e"} Nov 02 19:16:15.574633 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.574272 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx" event={"ID":"be154b44-f772-40b2-b398-630f2c47640d","Type":"ContainerDied","Data":"60708c54d62666c78855a1ecf80948f58c107b91825e1b4c67a201b3b18ff081"} Nov 02 19:16:15.574633 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.574301 5647 scope.go:122] "RemoveContainer" containerID="19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e" Nov 02 19:16:15.583833 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.583807 5647 scope.go:122] "RemoveContainer" containerID="19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e" Nov 02 19:16:15.584161 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:16:15.584139 5647 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e\": container with ID starting with 19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e not found: ID does not exist" containerID="19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e" Nov 02 19:16:15.584249 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.584169 5647 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e"} err="failed to get container status \"19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e\": rpc error: code = NotFound desc = could not find container \"19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e\": container with ID starting with 19165f8a41d98de76d4719cfd47afa6417f66bb7503ffe6c39b1702986ce4c7e not found: ID does not exist" Nov 02 19:16:15.740028 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.739986 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx"] Nov 02 19:16:15.790990 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:15.790959 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cb67577d6-7mnbx"] Nov 02 19:16:17.097516 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:17.097474 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be154b44-f772-40b2-b398-630f2c47640d" path="/var/lib/kubelet/pods/be154b44-f772-40b2-b398-630f2c47640d/volumes" Nov 02 19:16:18.136953 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:18.136906 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:16:18.137605 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:18.137568 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:16:18.492921 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:18.492830 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:16:18.493202 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:18.493170 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:16:35.947919 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:16:35.947880 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:16:38.743783 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:38.743727 5647 scope.go:122] "RemoveContainer" containerID="790296bb6fe0afaff780b73663058290880526ed6e7f7dd1c3f91f59966eed37" Nov 02 19:16:38.752695 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:38.752671 5647 scope.go:122] "RemoveContainer" containerID="37a11868aa7705a26839e0a3412113393e05028df23f31e6d5704ded1efe09fd" Nov 02 19:16:38.761568 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:38.761548 5647 scope.go:122] "RemoveContainer" containerID="8f8bca51abb646d2ecdb95fd8518efc7cea76eded26db890a2fe2bef02c665fc" Nov 02 19:16:46.055415 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:46.055376 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:16:58.157624 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:16:58.157589 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:17:05.946887 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:17:05.946850 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:17:19.540800 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:17:19.540755 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:17:31.756472 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:17:31.756432 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:17:35.946988 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:17:35.946945 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:18:04.378132 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:04.378043 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:18:05.946997 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:18:05.946963 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:18:15.244149 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:15.244112 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:18:27.961113 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:27.961069 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:18:35.156264 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:35.156232 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:18:35.168530 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:35.168500 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:18:35.946907 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:18:35.946863 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:18:36.045455 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:36.045417 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p"] Nov 02 19:18:36.095856 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:36.095821 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-t7j6p"] Nov 02 19:18:37.080113 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:37.080074 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e486ed1-cc19-415b-91ed-e821e2d88bf0" path="/var/lib/kubelet/pods/3e486ed1-cc19-415b-91ed-e821e2d88bf0/volumes" Nov 02 19:18:37.255150 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:18:37.255118 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:19:00.174858 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:00.174818 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:19:03.137412 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:03.136841 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:19:03.137412 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:03.136936 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:19:05.946592 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:19:05.946544 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:19:10.938034 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:10.938001 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:19:18.138916 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:18.138864 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:19:18.139584 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:18.139545 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:19:18.486821 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:18.483948 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:19:18.486821 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:18.484052 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:19:20.537510 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:20.537469 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:19:36.066760 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:19:36.065994 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:19:37.375799 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:37.375726 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:19:37.376275 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:37.375829 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:19:38.835285 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:38.835247 5647 scope.go:122] "RemoveContainer" containerID="b38eb8adada7222422004d653ffc34eac75c26d5e96eb3b079b09c32da1b723c" Nov 02 19:19:38.844867 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:38.844839 5647 scope.go:122] "RemoveContainer" containerID="838b5a2d94da9fa1d176081c8eefa0cf05b66a8684927cb9b32656299f0175e2" Nov 02 19:19:40.069920 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:19:40.069883 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:20:04.054835 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:04.054802 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:20:06.104996 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:20:06.104950 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:20:27.049487 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:27.049443 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw"] Nov 02 19:20:27.099956 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:27.099913 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-glclw"] Nov 02 19:20:29.084874 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:29.084834 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc66f43-024d-4b5b-aae9-6f79278d5661" path="/var/lib/kubelet/pods/6cc66f43-024d-4b5b-aae9-6f79278d5661/volumes" Nov 02 19:20:35.946638 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:20:35.946596 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:20:48.489947 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:48.489842 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:20:48.490598 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:48.490561 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:20:49.055759 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:49.047203 5647 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs"] Nov 02 19:20:49.151761 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:49.140094 5647 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-5d5cs"] Nov 02 19:20:51.080156 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:20:51.080123 5647 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44bca6b-b1e5-47ce-9f8d-cd72e440117c" path="/var/lib/kubelet/pods/b44bca6b-b1e5-47ce-9f8d-cd72e440117c/volumes" Nov 02 19:21:05.947206 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:21:05.947156 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:21:35.946053 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:21:35.946003 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:21:38.877995 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:21:38.877956 5647 scope.go:122] "RemoveContainer" containerID="34c6cd88743247b9ae82919ec0de6e2f3b6528564f2fad8c0236d404b8fb985c" Nov 02 19:21:52.400897 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:21:52.400851 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:21:52.401681 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:21:52.401635 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-bn7pt" podUID="42e84755-3ca6-4b00-8eca-2f8c7cfeb744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:21:52.610343 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:21:52.610307 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:21:52.610660 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:21:52.610617 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-5zzpw" podUID="f33d9232-9eef-4f5e-9240-ff28e8d84f96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:22:01.135481 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:01.135449 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-dzrt7"] Nov 02 19:22:03.179765 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:03.177896 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-26plb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:22:03.179765 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:03.179096 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-26plb" podUID="7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:22:03.487818 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:03.485861 5647 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 19:22:03.487818 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:03.485950 5647 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-4qpb8" podUID="237ddf20-334e-4c7e-8b07-6750aaa360b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 19:22:05.947119 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:22:05.947074 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:22:29.974044 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:29.974012 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fd65b9f5-t84fb_0af76160-9759-4f5e-b760-4a7210a7c8c7/manager/0.log" Nov 02 19:22:30.751542 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:30.751507 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b6969_6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b/proxy/0.log" Nov 02 19:22:31.736282 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:31.736223 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5c87d67b66-lkmpv_123a44a0-c06c-4846-bc41-2359c7cfee38/maas-api/0.log" Nov 02 19:22:35.817024 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:35.816972 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rblzl_c1cb0815-f24d-4ace-a544-ef0c90607758/postgres/0.log" Nov 02 19:22:35.946898 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:22:35.946862 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:22:38.898262 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:38.898226 5647 scope.go:122] "RemoveContainer" containerID="06a9526faa3e7705e29b7232da96fc9b3ba923d234116ff958cb5d9faa56ab54" Nov 02 19:22:39.245367 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:39.245282 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dzrt7_069d21b8-f43f-465f-870f-1da8bfc01136/limitador/0.log" Nov 02 19:22:40.691786 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:40.691751 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-svpxg_3d22bc9a-c7e3-47c2-acef-81df6d69ffa9/discovery/0.log" Nov 02 19:22:41.134103 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:41.134074 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87ddd77c-wgvlq_1e314e9c-09e2-41fd-a44c-0cc6e5213d11/kube-auth-proxy/0.log" Nov 02 19:22:41.668979 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:41.668951 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-26plb_7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607/istio-proxy/0.log" Nov 02 19:22:41.903341 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:41.903315 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-4qpb8_237ddf20-334e-4c7e-8b07-6750aaa360b9/istio-proxy/0.log" Nov 02 19:22:42.136511 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:42.136473 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-5zzpw_f33d9232-9eef-4f5e-9240-ff28e8d84f96/istio-proxy/0.log" Nov 02 19:22:42.644063 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:42.644022 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-bn7pt_42e84755-3ca6-4b00-8eca-2f8c7cfeb744/istio-proxy/0.log" Nov 02 19:22:44.148112 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:44.148079 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5fc8898f46-mmzfw_743fb649-948e-4a93-80d3-be11fd66e728/payload-pre-processing/0.log" Nov 02 19:22:46.569221 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:46.569187 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8x2zpd_2a224289-7a58-42a9-ae19-14b3050f9099/main/0.log" Nov 02 19:22:46.775666 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:22:46.775620 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-hd9vk_c622747f-8ec4-45a4-85af-c79f65ba4d13/main/0.log" Nov 02 19:23:05.947897 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:23:05.947850 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:23:07.892066 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:07.892028 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pxq56_bcf5ced3-8a35-47bb-a042-f7bfb1b5d60e/global-pull-secret-syncer/0.log" Nov 02 19:23:11.503129 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:11.503102 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x89xb_45f8e074-1aca-444b-91e3-5f5bc339164f/konnectivity-agent/0.log" Nov 02 19:23:11.894617 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:11.894581 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-71.us-west-1.compute.internal_a8894bf200d7f2fbf8f4d4549108d68f/haproxy/0.log" Nov 02 19:23:12.703270 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:12.703237 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fd65b9f5-t84fb_0af76160-9759-4f5e-b760-4a7210a7c8c7/manager/0.log" Nov 02 19:23:15.147023 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:15.146995 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b6969_6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b/proxy/0.log" Nov 02 19:23:18.596054 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.596022 5647 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl"] Nov 02 19:23:18.596488 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.596475 5647 cpu_manager.go:414] "containerMap: removing container" podUID="be154b44-f772-40b2-b398-630f2c47640d" containerName="maas-api" Nov 02 19:23:18.596535 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.596488 5647 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be154b44-f772-40b2-b398-630f2c47640d" containerName="maas-api" Nov 02 19:23:18.596577 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.596567 5647 memory_manager.go:365] "RemoveStaleState removing state" podUID="be154b44-f772-40b2-b398-630f2c47640d" containerName="maas-api" Nov 02 19:23:18.599676 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.599661 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.653118 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.653086 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdnvs\"/\"openshift-service-ca.crt\"" Nov 02 19:23:18.669859 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.669827 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-sys\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.670058 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.669909 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgsl2\" (UniqueName: \"kubernetes.io/projected/17e2c36c-1958-4550-947e-768e29d58bb2-kube-api-access-jgsl2\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.670058 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.669996 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-proc\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.670058 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.670039 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-lib-modules\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.670178 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.670110 5647 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-podres\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.703707 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.703674 5647 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pdnvs\"/\"default-dockercfg-hdbkh\"" Nov 02 19:23:18.704010 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.703943 5647 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdnvs\"/\"kube-root-ca.crt\"" Nov 02 19:23:18.705523 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.705499 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl"] Nov 02 19:23:18.770686 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770647 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-proc\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.770912 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770693 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-lib-modules\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.770912 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770765 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-podres\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.770912 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770809 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-proc\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.770912 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770834 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-sys\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.770912 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770866 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-lib-modules\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.770912 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770877 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-podres\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.771126 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770927 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17e2c36c-1958-4550-947e-768e29d58bb2-sys\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.771126 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.770960 5647 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgsl2\" (UniqueName: \"kubernetes.io/projected/17e2c36c-1958-4550-947e-768e29d58bb2-kube-api-access-jgsl2\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.828304 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.828271 5647 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgsl2\" (UniqueName: \"kubernetes.io/projected/17e2c36c-1958-4550-947e-768e29d58bb2-kube-api-access-jgsl2\") pod \"perf-node-gather-daemonset-gcgsl\" (UID: \"17e2c36c-1958-4550-947e-768e29d58bb2\") " pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:18.912216 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:18.912128 5647 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:19.283407 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:19.283382 5647 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl"] Nov 02 19:23:19.669483 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:19.669444 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" event={"ID":"17e2c36c-1958-4550-947e-768e29d58bb2","Type":"ContainerStarted","Data":"486a8e3d0a524bbe8c780ae748679f932ddfd8931bf535458ab99d73ea260cee"} Nov 02 19:23:19.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:19.669505 5647 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" event={"ID":"17e2c36c-1958-4550-947e-768e29d58bb2","Type":"ContainerStarted","Data":"4daf47b1b4edebdfde498b6ecf6aa2f29074c05a4336bdfe2b471bf31af8654a"} Nov 02 19:23:19.669935 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:19.669532 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:19.785548 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:19.785491 5647 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" podStartSLOduration=1.785478732 podStartE2EDuration="1.785478732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 19:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 19:23:19.784247829 +0000 UTC m=+7186.857775044" watchObservedRunningTime="2026-11-02 19:23:19.785478732 +0000 UTC m=+7186.859005935" Nov 02 19:23:20.268652 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:20.268600 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fd65b9f5-t84fb_0af76160-9759-4f5e-b760-4a7210a7c8c7/manager/0.log" Nov 02 19:23:20.388946 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:20.388912 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-dzrt7_069d21b8-f43f-465f-870f-1da8bfc01136/limitador/0.log" Nov 02 19:23:22.730701 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:22.730665 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b6969_6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b/proxy/0.log" Nov 02 19:23:23.580193 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:23.580161 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/alertmanager/0.log" Nov 02 19:23:23.995108 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:23.995075 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/config-reloader/0.log" Nov 02 19:23:24.412750 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:24.412704 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/kube-rbac-proxy-web/0.log" Nov 02 19:23:24.843306 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:24.843279 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/kube-rbac-proxy/0.log" Nov 02 19:23:25.259832 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:25.259751 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/kube-rbac-proxy-metric/0.log" Nov 02 19:23:25.675368 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:25.675335 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/prom-label-proxy/0.log" Nov 02 19:23:25.685171 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:25.685147 5647 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pdnvs/perf-node-gather-daemonset-gcgsl" Nov 02 19:23:26.092290 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:26.092264 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fbd45d5-6de4-4bb2-ba5f-f8e6eefc48c2/init-config-reloader/0.log" Nov 02 19:23:26.939807 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:26.939779 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7glpq_361abe17-e8b7-4695-ae30-8cd0ccc06017/kube-state-metrics/0.log" Nov 02 19:23:27.353415 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:27.353388 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7glpq_361abe17-e8b7-4695-ae30-8cd0ccc06017/kube-rbac-proxy-main/0.log" Nov 02 19:23:27.363950 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:27.363929 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7fd65b9f5-t84fb_0af76160-9759-4f5e-b760-4a7210a7c8c7/manager/0.log" Nov 02 19:23:27.778381 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:27.778298 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7glpq_361abe17-e8b7-4695-ae30-8cd0ccc06017/kube-rbac-proxy-self/0.log" Nov 02 19:23:28.200781 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:28.200729 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f564b5ffb-5scd2_809d0903-74ab-4c73-a6db-5811776a58ec/metrics-server/0.log" Nov 02 19:23:28.621705 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:28.621672 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-gjj99_a924f2fb-68fd-4d19-bb5d-26eb25dcd9bd/monitoring-plugin/0.log" Nov 02 19:23:29.879817 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:29.879782 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-b6969_6a605b3e-d7f1-4af2-89aa-e0063bfb2c7b/proxy/0.log" Nov 02 19:23:31.557682 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:31.557656 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jmf49_9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/node-exporter/0.log" Nov 02 19:23:31.971545 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:31.971514 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jmf49_9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/kube-rbac-proxy/0.log" Nov 02 19:23:32.386722 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:32.386694 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jmf49_9a23d348-6b2d-4ded-9ff9-543a1e01f6cf/init-textfile/0.log" Nov 02 19:23:32.806908 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:32.806826 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lbz97_d15153aa-78ce-426a-9b63-5a4489d8fc45/kube-rbac-proxy-main/0.log" Nov 02 19:23:33.223006 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:33.222973 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lbz97_d15153aa-78ce-426a-9b63-5a4489d8fc45/kube-rbac-proxy-self/0.log" Nov 02 19:23:33.638150 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:33.638091 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lbz97_d15153aa-78ce-426a-9b63-5a4489d8fc45/openshift-state-metrics/0.log" Nov 02 19:23:34.068153 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:34.068067 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/prometheus/0.log" Nov 02 19:23:34.483004 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:34.482971 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/config-reloader/0.log" Nov 02 19:23:34.899540 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:34.899495 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/thanos-sidecar/0.log" Nov 02 19:23:35.199588 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:35.199501 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:23:35.212225 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:35.212193 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-67tdm_63987b50-f8b5-42a2-a5cf-50fc6cec635e/ovn-acl-logging/0.log" Nov 02 19:23:35.317290 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:35.317257 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/kube-rbac-proxy-web/0.log" Nov 02 19:23:35.546656 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:35.546572 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-svpxg_3d22bc9a-c7e3-47c2-acef-81df6d69ffa9/discovery/0.log" Nov 02 19:23:35.733845 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:35.733809 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/kube-rbac-proxy/0.log" Nov 02 19:23:35.866982 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:35.866945 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87ddd77c-wgvlq_1e314e9c-09e2-41fd-a44c-0cc6e5213d11/kube-auth-proxy/0.log" Nov 02 19:23:35.948592 ip-10-0-128-71 kubenswrapper[5647]: E1102 19:23:35.948554 5647 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmkz5" containerName="extractor" Nov 02 19:23:36.151503 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:36.151430 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/kube-rbac-proxy-thanos/0.log" Nov 02 19:23:36.304015 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:36.303983 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-26plb_7ebbc3ec-4bd5-40fb-8ce6-338f3b4b7607/istio-proxy/0.log" Nov 02 19:23:36.516728 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:36.516641 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-4qpb8_237ddf20-334e-4c7e-8b07-6750aaa360b9/istio-proxy/0.log" Nov 02 19:23:36.568257 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:36.568227 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2d999266-4195-4a21-a31e-e832f991e728/init-config-reloader/0.log" Nov 02 19:23:36.731643 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:36.731610 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-5zzpw_f33d9232-9eef-4f5e-9240-ff28e8d84f96/istio-proxy/0.log" Nov 02 19:23:36.992466 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:36.992435 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-m2bk8_34b5f718-d276-44f2-b1b5-f9bf961147ed/prometheus-operator/0.log" Nov 02 19:23:37.163102 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:37.163066 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-bn7pt_42e84755-3ca6-4b00-8eca-2f8c7cfeb744/istio-proxy/0.log" Nov 02 19:23:37.402099 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:37.402072 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-m2bk8_34b5f718-d276-44f2-b1b5-f9bf961147ed/kube-rbac-proxy/0.log" Nov 02 19:23:37.821288 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:37.821211 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-lwmxf_c31ce7db-987c-4f05-9a4b-fbfd2fa0b113/prometheus-operator-admission-webhook/0.log" Nov 02 19:23:38.249138 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:38.249109 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8694f5cb-nt4tw_391e59bd-0f60-41e7-a7db-54f3bf5eddd3/telemeter-client/0.log" Nov 02 19:23:38.442954 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:38.442926 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-5fc8898f46-mmzfw_743fb649-948e-4a93-80d3-be11fd66e728/payload-pre-processing/0.log" Nov 02 19:23:38.664417 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:38.664385 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8694f5cb-nt4tw_391e59bd-0f60-41e7-a7db-54f3bf5eddd3/reload/0.log" Nov 02 19:23:39.081075 ip-10-0-128-71 kubenswrapper[5647]: I1102 19:23:39.080996 5647 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5f8694f5cb-nt4tw_391e59bd-0f60-41e7-a7db-54f3bf5eddd3/kube-rbac-proxy/0.log"